Timeline for Can non-differentiable layer be used in a neural network, if it's not learned?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 5, 2022 at 1:19 | comment | added | user76284 | math.stackexchange.com/a/2885265 | |
| Jan 9, 2019 at 1:50 | comment | added | NicNic8 | @NeilSlater I very much appreciate your efforts here. I also think it's important to be careful when saying things like "it is not even very important in this case if we hit a non-differentiable point". Many theorems regarding the convergence and the rate of convergence of gradient descent aren't valid unless the objective function is differentiable. | |
| Jan 8, 2019 at 21:53 | comment | added | littleO | Thanks. I think it's worth being careful about this point, because there are examples where gradient descent (with exact like search) fails to converge to a local minimizer despite never encountering a nondifferentiable point along the way. (Such an example can be found in Vandenberghe's 236c notes.) So, it seems to me that there is a serious theory question about how well we can expect gradient descent to perform when the objective is nondifferentiable, even if the objective is differentiable almost everywhere. If the goal is to grock deep learning, I think this point should be grappled with. | |
| Jan 8, 2019 at 20:19 | comment | added | Neil Slater | @NicNic8: Yes, but note that the question here is not really about the maths - the OP thought that the function was not differentiable at all, whilst it is entirely possible to use the chain rule in domains of the input functions that are differentiable. Further to that, it is not even very important in this case if we hit a non-differentiable point, we can safely patch it. That may not be the case for all functions or for all situations where you want to use the chain rule . . . but that's a different question | |
| Jan 8, 2019 at 20:12 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 305 characters in body
|
| Jan 8, 2019 at 20:09 | comment | added | NicNic8 | This answer is related to the math.stackexchange question here: math.stackexchange.com/questions/2837737/… | |
| Jan 8, 2019 at 20:08 | comment | added | Neil Slater | @littleO: Those sharp points are not important to gradient descent by backpropagation in practice, and you can take whichever of the two overlapping values you prefer as the value of the "gradient" - everything will work fine. I will try to add that detail somehow without changing the flow of the answer. | |
| Jan 8, 2019 at 17:00 | comment | added | littleO | To describe a function as differentiable means that it is differentiable at all points in its domain, and this is not true of the max-pooling function. If $f(x) = \max_i f_i(x)$, then $f$ is typically not a differentiable function, even if each function $f_i$ is. If we look at the graph of $f$ we will see some sharp corners. | |
| Aug 24, 2018 at 22:07 | vote | accept | RedRus | ||
| Aug 24, 2018 at 18:58 | history | answered | Neil Slater | CC BY-SA 4.0 |