1
$\begingroup$

In this research paper, we have the following claim

the smoothness assumption that underlies many kernel methods such as Support Vector Machines (SVMs) does not hold for deep neural networks trained through backpropagation

Does smoothness here refer to no sharp rise/fall in gradients?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

Smoothness here is the mathematical definition, so as you implied smoothness is ruled out by output data with sharp spikes or discontinuous jumps (and possibly the data of the gradient, the gradient's gradient, ad infinitum, depending on who defines smoothness).

By any definition a lot of activation functions are not smooth, for example RELU. This means neural networks in general are not smooth.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .