I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.
$\begingroup$
$\endgroup$
Add a comment
|
$\begingroup$
$\endgroup$
4
By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for tf.nn.relu. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.
In a way this is compatible with the Python philosophy that explicit is better than implicit. If you mean to use $\mathrm{ReLU}$, it's probably best to use tf.nn.relu if it is suitable for your use case.
[1] Vinod Nair and Geoffrey Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML'10 (2010). URL.
-
1$\begingroup$ Thanks a lot. That makes sense. Do you know then what is the easiest way to convert tf.abs into differentiable operation with a gradient at 0 defined as 0? $\endgroup$ – zedsdead Feb 17 at 22:25
-
$\begingroup$ @zedsdead Could you just substitute in
tf.nn.reluin the place you're usingtf.abs? $\endgroup$ – htl Feb 18 at 9:45 -
2
-
$\begingroup$ @htl Well, they are not the same thing but NikoNyrh’s answer below is probably the easiest solution. $\endgroup$ – zedsdead Feb 23 at 16:36
$\begingroup$
$\endgroup$
Creating custom gradient for tf.abs may solve the problem:
@tf.custom_gradient
def abs_with_grad(x):
y = tf.abs(x);
def grad(div): # Derivation intermediate value
g = 1; # Use 1 to make the chain rule just skip abs
return div*g;
return y,grad;
