For example, AFAIK, the pooling layer in a CNN is not differentiable, but it can be used because it's not learning. Is it always true?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so.
For example:
If an average pooling layer has inputs $z$ and outputs $a$, and each output is average of 4 inputs then $\frac{da}{dz} = 0.25$ (if pooling layers overlap it gets a little more complicated, but you just add things up where they overlap).
A max pooling layer has $\frac{da}{dz} = 1$ for the maximum z, and $\frac{da}{dz} = 0$ for all others.
A pooling layer usually has no learnable parameters, but if you know the gradient of a function at its outputs, you can assign gradient correctly to its inputs using the chain rule. That is essentially all that back propagation is, the chain rule applied to the functions of a neural network.
To answer your question more directly:
Can non-differentiable layer be used in a neural network, if it's not learned?
No.
There is one exception: If this layer appears directly after the input, then as it has no parameters to learn, and you generally do not care about the gradient of the input data, so you can have a non-differentiable function there. However, this is just the same as transforming your input data in some non-differentiable way, and training the NN with that transformed data instead.
* Technically there are some discontinuities in the gradient of a max function (where any two inputs are equal). However, this is not a problem in practice, as the gradients are well behaved close to these values. When you can safely do this or not is probably the topic of another question.
answered Aug 24, 2018 at 18:58
-
1$\begingroup$ To describe a function as differentiable means that it is differentiable at all points in its domain, and this is not true of the max-pooling function. If $f(x) = \max_i f_i(x)$, then $f$ is typically not a differentiable function, even if each function $f_i$ is. If we look at the graph of $f$ we will see some sharp corners. $\endgroup$– littleOCommented Jan 8, 2019 at 17:00
-
$\begingroup$ @littleO: Those sharp points are not important to gradient descent by backpropagation in practice, and you can take whichever of the two overlapping values you prefer as the value of the "gradient" - everything will work fine. I will try to add that detail somehow without changing the flow of the answer. $\endgroup$ Commented Jan 8, 2019 at 20:08
-
$\begingroup$ This answer is related to the math.stackexchange question here: math.stackexchange.com/questions/2837737/… $\endgroup$– NicNic8Commented Jan 8, 2019 at 20:09
-
2$\begingroup$ Thanks. I think it's worth being careful about this point, because there are examples where gradient descent (with exact like search) fails to converge to a local minimizer despite never encountering a nondifferentiable point along the way. (Such an example can be found in Vandenberghe's 236c notes.) So, it seems to me that there is a serious theory question about how well we can expect gradient descent to perform when the objective is nondifferentiable, even if the objective is differentiable almost everywhere. If the goal is to grock deep learning, I think this point should be grappled with. $\endgroup$– littleOCommented Jan 8, 2019 at 21:53
-
1$\begingroup$ @NeilSlater I very much appreciate your efforts here. I also think it's important to be careful when saying things like "it is not even very important in this case if we hit a non-differentiable point". Many theorems regarding the convergence and the rate of convergence of gradient descent aren't valid unless the objective function is differentiable. $\endgroup$– NicNic8Commented Jan 9, 2019 at 1:50