All the activation functions I see have positive derivatives. Will negative ReLU work as well as its positive counterpart or will it lead to instability?


1 Answer 1


By definition from wikipedia:

Let $\varphi$ be a nonconstant, bounded, and continuous function.

This is also being proven for ReLU and other functions that violets some of those properties... however, if you take 1 - sigmoid(x) as your activation, all the properties are satisfied, and the derivative is strictly negative

This can be more "popularly" be seen also in the max-out activation function


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .