In this research paper, we have the following claim

the smoothness assumption that underlies many kernel methods such as Support Vector Machines (SVMs) does not hold for deep neural networks trained through backpropagation

Does smoothness here refer to no sharp rise/fall in gradients?


Smoothness here is the mathematical definition, so as you implied smoothness is ruled out by output data with sharp spikes or discontinuous jumps (and possibly the data of the gradient, the gradient's gradient, ad infinitum, depending on who defines smoothness).

By any definition a lot of activation functions are not smooth, for example RELU. This means neural networks in general are not smooth.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.