All the activation functions I see have positive derivatives. Will negative ReLU work as well as its positive counterpart or will it lead to instability?
1 Answer
$\begingroup$
$\endgroup$
By definition from wikipedia:
Let $\varphi$ be a nonconstant, bounded, and continuous function.
This is also being proven for ReLU and other functions that violets some of those properties... however, if you take 1 - sigmoid(x) as your activation, all the properties are satisfied, and the derivative is strictly negative
This can be more "popularly" be seen also in the max-out activation function
