How does gradient descent work with relu if weights are negative?

Question

How does gradient descent work with relu, imagine the weights are quite negative and so our "prediction" is 0, then not much is learned. Is there a risk that training gets stuck when weights start negative or small?

Answer 1

The issue you have described is called the dying ReLU, which is basically about getting a gradient of zero when negative predicted values got thresholded to zero.

In general this is only an issue when ALL the units in a layer (also for all layers) predict negative values. So only in this extreme situation your network won't learn anything because the derivative is zero.

But it can happen that some units in a Dense layer (for example) always output a zero, in this case these have died and so they are not useful anymore for learning.

The way to fix the issue, is to change activation function (but I guess that weight initialization may also help) to something like: leaky ReLU (which introduces a negative slope where the gradient exists), ELU (exponential linear unit; slower to compute but never dies), or even SELU (scaled ELU).