I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.


2 Answers 2


By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for tf.nn.relu. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.

In a way this is compatible with the Python philosophy that explicit is better than implicit. If you mean to use $\mathrm{ReLU}$, it's probably best to use tf.nn.relu if it is suitable for your use case.

[1] Vinod Nair and Geoffrey Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML'10 (2010). URL.

  • 1
    $\begingroup$ Thanks a lot. That makes sense. Do you know then what is the easiest way to convert tf.abs into differentiable operation with a gradient at 0 defined as 0? $\endgroup$
    – zedsdead
    Feb 17, 2021 at 22:25
  • $\begingroup$ @zedsdead Could you just substitute in tf.nn.relu in the place you're using tf.abs? $\endgroup$
    – htl
    Feb 18, 2021 at 9:45
  • 2
    $\begingroup$ How about relu(x) + relu(-x)? :) $\endgroup$
    – NikoNyrh
    Feb 20, 2021 at 9:47
  • $\begingroup$ @htl Well, they are not the same thing but NikoNyrh’s answer below is probably the easiest solution. $\endgroup$
    – zedsdead
    Feb 23, 2021 at 16:36

Creating custom gradient for tf.abs may solve the problem:

def abs_with_grad(x):
  y = tf.abs(x);

  def grad(div): # Derivation intermediate value
    g = 1; # Use 1 to make the chain rule just skip abs
    return div*g;

  return y,grad;

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .