New answers tagged artificial-neuron
0
I think a nice back-of-the envelope calculation is the intuition for exploding/vanishing gradients in RNNs:
Simplifications: diagonalisable weights, no non-linearities, 1 layer: $h_t = W\cdot h_{t-1} + U\cdot x_t$
Let $L_t$ be the loss at timestep $t$ with the total loss $L = \sum_t L_t$
$$
\frac{\partial L_t}{\partial W} \sim= \sum_{k=1}^{t}\frac{\...
Top 50 recent answers are included
