1
$\begingroup$

I am currently looking into LSTMs. I found this nice blog post, which is already very helpful, but still, there are things I don't understand, mostly because of the collapsed layers.

  • The input $X_t$, and the output of the previous time step $H_{t-1}$, how do they get combined? Multiplied, added or what?
  • The input weights and the weights of the input of the previous time step, those are just the weights of the connections between the time-steps/units, right?
$\endgroup$
  • $\begingroup$ All i would like to say head over to this video youtube.com/… by Andrew Ng you will understand everything $\endgroup$ – DuttaA Sep 2 '18 at 16:28
1
$\begingroup$

(1) $X_t$ and $H_{t-1}$ are concatenated. The blog you cited explained its notation "Lines merging denote concatenation". For example, if $X_t=[1,2,3]$ and $H_{t-1}=[4,5,6,7]$, then their concatenation is $[1,2,3,4,5,6,7]$

(2) When you say "input weights" or "weights of the input of the previous time step", are you referring to the $W_i$ in your cited blog? If so they are not the weights of the connections between the time-steps/units. They are part of the input gate only. The connections between the time-steps/units do not have weights applied to them.

$\endgroup$
  • $\begingroup$ Thanks for the answer. But: If on every time step, more elements are appended, doesn't the vector get huge after many time steps? I mean, there is the forget, ignore and update gate, but as far as I know, they only set the values they want to block to zero, they don't erase the value from the vector. $\endgroup$ – Ben Sep 3 '18 at 10:26
  • $\begingroup$ So each gate is essentially a small NN, just collapsed down into one unit using matrix multiplication? $\endgroup$ – Ben Sep 3 '18 at 17:39
  • $\begingroup$ @Ben In each step $X_t$ and $H_{t-1}$ is fixed size, so it won't become longer and longer. Note that in each cell, the concatenation of $H_{t-1}$ and $X_t$ are not directly connected to $H_{t}$, but they go through a $\sigma$ which "collapse down" the size. And yes, here $\sigma$ is small NN. $\endgroup$ – user12075 Sep 3 '18 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.