Consider the following statement (from the paper Generative Adversarial Nets) about the success of discriminative models
So far, the most striking successes in deep learning have involved discriminative models, usually those that map a high-dimensional, rich sensory input to a class label. These striking successes have primarily been based on the backpropagation and dropout algorithms, using piecewise linear units which have a particularly well-behaved gradient.
The piece-wise linear units they are referring to are, I guess, the activation functions. The primary purpose of activation functions is to introduce non-linearity, and there are no other mathematical requirements, such as continuity, differentiability, etc. But not all the activation functions may work well, and some are preferred over others, based on their nature, as well as the task under consideration.
After reading the quoted paragraph, one can conclude that the activation functions that have well-behaved gradient are showing better results than the others, at least in discriminative tasks.
What does "well-behaved" in this context stand for? Can we have some mathematical properties of the gradient in order to recognize it as a well-behaved gradient? Or the usage of the phrase "well-behaved" is highly dependent on the discriminative task under consideration?
