Timeline for What are the conceptual differences between regularisation and optimisation in deep neural nets?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 27, 2020 at 12:18 | vote | accept | Felipe Martins Melo | ||
| Nov 26, 2020 at 20:41 | comment | added | nbro | Maybe this answer will also be useful, especially one of the videos by Hinton that I am linking to there. Btw, in the case of dropout, we can think that regularisation is happening before updating the weights, the drop of the units happens before computing the output of the NN. | |
| Nov 26, 2020 at 20:37 | comment | added | nbro | @FelipeMartinsMelo You could do that, but you often do both at the same time, i.e. you include a "regularisation term" in your objective function, then you optimize this "regularized objective function". These explanations apply more to $L_1$ and $L_2$ regularisation than to dropout or other ad-hoc techniques. $L_1$ and $L_2$ regularisation are theoretically motivated as a "Bayesian way" of regularizing the weights (e.g. assuming that the weights are sampled from a normal distribution). You can actually derive $L_2$ regularisation from Bayes' rule by putting a Gaussian on the weights. | |
| Nov 26, 2020 at 20:34 | comment | added | Felipe Martins Melo | thank you. From your explanation that optimisation tries to identify the best parameters and regularisation tries to constrain them, is it correct to state that they can always work together as long as optimisation happens first? in other words, i first identify the parameters and them constrain them. Makes any sense? | |
| Nov 26, 2020 at 19:05 | history | edited | nbro | CC BY-SA 4.0 |
deleted 10 characters in body
|
| Nov 26, 2020 at 18:59 | history | edited | nbro | CC BY-SA 4.0 |
deleted 10 characters in body
|
| Nov 26, 2020 at 18:51 | history | answered | nbro | CC BY-SA 4.0 |