relu inequality

Suppose we want a transformer to evaluate the inequality

returning $c=1$ if $x\le 0$ and $c=0$ otherwise. For integer $x$, this can be done with a feedforward ReLU network computing

Observe that for $x \ge 1$ this is simply $c = 1 - x + (x - 1) = 0$. For $x \le 0$ this is $1 - 0 + 0 = 1$. Thus this expression computes the inequality.

If $x$ fails to be an integer and we have $0 < x < 1$, then the expression evaluates to $c = 1 - x$, which smoothly interpolates between the discrete outcomes of $1$ and $0$.