1
$\begingroup$

I am currently looking into LSTMs. I found this nice blog post, which is already very helpful, but still, there are things I don't understand, mostly because of the collapsed layers.

  • The input $X_t$, and the output of the previous time step $H_{t-1}$, how do they get combined? Multiplied, added or what?
  • The input weights and the weights of the input of the previous time step, those are just the weights of the connections between the time-steps/units, right?
Improve this question
$\endgroup$
1
  • $\begingroup$ All i would like to say head over to this video youtube.com/… by Andrew Ng you will understand everything $\endgroup$
    – user9947
    Sep 2, 2018 at 16:28

1 Answer 1

Reset to default
1
$\begingroup$

(1) $X_t$ and $H_{t-1}$ are concatenated. The blog you cited explained its notation "Lines merging denote concatenation". For example, if $X_t=[1,2,3]$ and $H_{t-1}=[4,5,6,7]$, then their concatenation is $[1,2,3,4,5,6,7]$

(2) When you say "input weights" or "weights of the input of the previous time step", are you referring to the $W_i$ in your cited blog? If so they are not the weights of the connections between the time-steps/units. They are part of the input gate only. The connections between the time-steps/units do not have weights applied to them.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer. But: If on every time step, more elements are appended, doesn't the vector get huge after many time steps? I mean, there is the forget, ignore and update gate, but as far as I know, they only set the values they want to block to zero, they don't erase the value from the vector. $\endgroup$
    – Ben
    Sep 3, 2018 at 10:26
  • $\begingroup$ So each gate is essentially a small NN, just collapsed down into one unit using matrix multiplication? $\endgroup$
    – Ben
    Sep 3, 2018 at 17:39
  • $\begingroup$ @Ben In each step $X_t$ and $H_{t-1}$ is fixed size, so it won't become longer and longer. Note that in each cell, the concatenation of $H_{t-1}$ and $X_t$ are not directly connected to $H_{t}$, but they go through a $\sigma$ which "collapse down" the size. And yes, here $\sigma$ is small NN. $\endgroup$
    – user12075
    Sep 3, 2018 at 18:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .