3
$\begingroup$

While looking into the vanishing gradient problem, I came across a paper (https://ieeexplore.ieee.org/abstract/document/9336631) that used artificial derivatives in lieu of the real derivatives. For a visualization, see the attached image:

As you can see, the artificial derivative of the ReLU is the sigmoid function, which smoothes out the derivative to circumvent the dying ReLU problem. The authors apply similar smoothing operations for other activation functions such as the sigmoid. This idea seems pretty convincing, and it surprises me that it has not been used thus far, especially considering there is no additional computational cost in replacing the derivatives, as the authors claim. So why is it that vanishing gradients have not been addressed in this way in conventional deep learning frameworks such as Pytorch or Tensorflow?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

This idea seems pretty convincing

Indeed, you don't have to use the exact gradient of the activation function during the backward step.

The gradient of the activation function is a multiplicative part of the gradient of the loss. However, the exact gradient of the loss is almost never used. It is used neither in stochastic gradient descent, mini-batch, training with momentum, Adam, Newton nor in most other methods.

As only the approximate gradient of the loss is used in practice, we don't have to keep any part of it exact, so there is no need to keep the gradient of the activation function exact. It can be taken approximately, and any approximation is fine (including from the paper you referenced) as long as the loss decreases during training.

So why is it that vanishing gradients have not been addressed in this way in conventional deep learning frameworks such as Pytorch or Tensorflow?

This can be done in PyTorch and TensorFlow. You can modify the gradient by specifying the backward() method in PyTorch or grad() method in TensorFlow:

Extending PyTorch with Custom Activation Functions

tf.custom_gradient

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your reply. Your point about SGD makes sense to me. I also appreciate you mentioning how one can manually adapt the backward/grad method. However, I think you did not directly answer my second question: why are artificial derivatives not used as much? Is it because people simply use batch normalisation when they observe vanishing gradients? $\endgroup$
    – postnubilaphoebus
    Commented Oct 11, 2022 at 12:01
  • 1
    $\begingroup$ I am not sure they are not used. I showed that it's easy to use them with popular frameworks, although batch norm is even easier in TensorFlow and PyTorch: 1 line of code instead of a few, maybe that's why. Previously, I encountered their usage in discussions of what to do with discontinuous activation functions (including in stackexchange and kaggle) long before the paper you refer to was published. If that problem can be solved by using approximate gradients instead of exact ones, I suppose that many other problems of activation functions including vanishing gradients can be solved similarly $\endgroup$
    – Vladislav Gladkikh
    Commented Oct 11, 2022 at 12:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .