New answers tagged backpropagation
0
Deep Learning by Goodfellow et. al is a good book for anything related neural networks, and it's freely available online. Backpropagation is covered in Chapter 6.5.
1
Creating custom gradient for tf.abs may solve the problem:
@tf.custom_gradient
def abs_with_grad(x):
y = tf.abs(x);
def grad(div): # Derivation intermediate value
g = 1; # Use 1 to make the chain rule just skip abs
return div*g;
return y,grad;
3
By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for tf.nn.relu. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an ...
Top 50 recent answers are included
