How does gradient descent work with ReLU if weights are negative?

Question

How does gradient descent work with ReLU, imagine the weights are quite negative and so our "prediction" is 0, then not much is learned. Is there a risk that training gets stuck when weights start negative or small?

Answer 1

The issue you have described is called the dying ReLU, which is basically about getting a gradient of zero when negative predicted values got thresholded to zero.

In general this is only an issue when ALL the units in a layer (also for all layers) predict negative values. So only in this extreme situation your network won't learn anything because the derivative is zero.

But it can happen that some units in a Dense layer (for example) always output a zero, in this case these have died and so they are not useful anymore for learning.

The way to fix the issue, is to change activation function (but I guess that weight initialization may also help) to something like: leaky ReLU (which introduces a negative slope where the gradient exists), ELU (exponential linear unit; slower to compute but never dies), or even SELU (scaled ELU).

Answer 2

Dying ReLU Problem sometimes occurs with Relu activation function.

Dying ReLU Problem:

• is during backpropagation, once the nodes(neurons) with ReLU activation function recieve negative input values, they always produce zero for any input values, finally, they are never recovered to produce any values except zero, then a model cannot be trained effectively.
• is also called Dead ReLU problem.
• more easily occurs with:
  • higher learning rates.
  • higher negative bias.
• can be detected if:
  • convergence is slow or stopped.
  • a loss function returns nan.
• can be mitigated using:
  • lower learning rate.
  • a positive bias.
  • Leaky ReLU activation function.
  • PReLU activation function.
  • ELU activation function.

Answer 3

Gradient Descent fails to updates weights due to Dying RELU problem resulting in poor learning, preventing the network to capture underlying patterns of the training data, leading to poor generalization on unseen data