New answers tagged

0

I think a nice back-of-the envelope calculation is the intuition for exploding/vanishing gradients in RNNs: Simplifications: diagonalisable weights, no non-linearities, 1 layer: $h_t = W\cdot h_{t-1} + U\cdot x_t$ Let $L_t$ be the loss at timestep $t$ with the total loss $L = \sum_t L_t$ $$ \frac{\partial L_t}{\partial W} \sim= \sum_{k=1}^{t}\frac{\...


Top 50 recent answers are included