0
$\begingroup$

Consider the following statement about the success of discriminative models

So far, the most striking successes in deep learning have involved discriminative models, usually those that map a high-dimensional, rich sensory input to a class label. These striking successes have primarily been based on the backpropagation and dropout algorithms, using piecewise linear units which have a particularly well-behaved gradient.

The piece-wise linear units they are referring are about are, i guess, the activation functions. The primary purpose of activation functions is to introduce non-linearity and there are no other mathematical requirements such as continuity, differentiability etc., But, all the activation functions may not work well and some are preferred over other based on their nature as well as the task under consideration.

After reading the quoted paragraph one can conclude that the activation functions that have well-behaved gradient are showing better results than the other at-least in discriminative tasks.

What does "well-behaved" in this context stand for? Can we have some mathematical properties of gradient in-order recognize it as a well-behaved gradient? Or the usage of the phrase "well-behaved" is higly dependent on the discriminative task under consideration?

$\endgroup$
0
$\begingroup$

"Piecewise linear" appears to describe ReLu and LRelu activation functions, whose gradients are just simple step functions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.