Timeline for What is the goal of weight initialization in neural networks?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 25, 2020 at 14:30 | vote | accept | S2673 | ||
| Sep 24, 2020 at 3:38 | answer | added | Saurav Maheshkar | timeline score: 4 | |
| Sep 22, 2020 at 16:58 | comment | added | nbro | This article Why Initialize a Neural Network with Random Weights? should be useful. | |
| Sep 22, 2020 at 16:50 | history | edited | S2673 | CC BY-SA 4.0 |
Added question
|
| Sep 22, 2020 at 0:21 | comment | added | S2673 | @Recessive I actually just read that, but thanks. I am not only considering random initialization. I am not really looking for a specific strategy, but more what a perfect one would be supposed to achieve. | |
| Sep 22, 2020 at 0:17 | comment | added | Recessive | It also depends on where the line is drawn. Transfer learning is a form of weight initialisation, and in that case that is an exceptional efficient initialiser. If you only consider random initialisation, I would suggest reading this: wandb.com/articles/…. It's quite comprehensive | |
| Sep 22, 2020 at 0:10 | comment | added | S2673 | @Recessive I’m using perfect very loosely and also I am not asking what the perfect weights would be, but what they would achieve in speeding up learning. | |
| Sep 21, 2020 at 23:24 | comment | added | Recessive | Finding the "perfect" anything of a black box model is not feasible. You can use a good initialisation method though. As an example, the goal of Xavier initialisation is to maintain a variance at the end of each layer equal to the incoming variance. This helps dramatically with keeping gradients reasonable | |
| Sep 21, 2020 at 23:15 | answer | added | Robby Goetschalckx | timeline score: 1 | |
| Sep 21, 2020 at 23:10 | history | asked | S2673 | CC BY-SA 4.0 |