From Deep Learning (Courville, Goodfellow, Bengio), a ReLU activation often "dies" because

One drawback to rectified linear units is that they cannot learn via gradient based methods on examples for which their activation is zero.

Similarly, L1 regularization (as opposed to L2) results in a sparse network

This demonstrates that L2 regularization does not cause the parameters to become sparse, while L1 regularization may do so for large enough α. The sparsity property induced by L1 regularization has been used extensively as a feature selection mechanism.

A couple questions about these topics:

  1. In practice, is there any way/use to prune these "dead" ReLU-activated neurons? And if our trained network performs well with lots of dead neurons, would that imply that a shallower network is a sufficient representation?
  2. Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization? If it does, does this then imply that sigmoid/tanh activations don't have this property?
  • $\begingroup$ Please, select one question and remove the others, which you can ask in a separate post! $\endgroup$
    – nbro
    Commented Jan 21, 2023 at 12:01

1 Answer 1

  • No you can't prune dead ReLUs without accuracy loss because even if you find that some ReLUs are dead for a small batch of inputs, you don't know if it's dead for other inputs. If you wanted to prune like this you'd need to do more finetuning. Pruning dead ReLUs will not make your network shallower obviously.

  • I'm pretty sure they're not mathematically equivalent.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .