4
$\begingroup$

What is the point of having multiple LSTM units in a single layer?

Surely if we have a single unit it should be able to capture (remember) all the data anyway and using more units in the same layer would just make the other units learn exactly the same historical features?

I've even shown myself empirically that using multiple LSTMs in a single layer improves performance, but in my head it still doesn't make sense, because I don't see what is it that other units are learning that others aren't? Is this sort of similar to how we use multiple filters in a single CNN layer?

$\endgroup$
2
  • $\begingroup$ Your assumption "Surely if we have a single unit it should be able to capture (remember) all the data anyway" is likely incorrect. Why do you think a single unit would be sufficient? This is just an interpretation, but, if you have to keep track of multiple concepts at the same time step, then a single RNN unit might not be sufficient. $\endgroup$ – nbro May 15 '19 at 22:15
  • $\begingroup$ that was my question i got here : ai.stackexchange.com/questions/12657/… : for RNN, multiple cells/units, whatever store the data from previous time stamps and use it for backpropagate. Just RNN cell store only weights abd biases, and LSTM stores more stuff what is in there $\endgroup$ – user8426627 Jun 15 '19 at 1:05
2
$\begingroup$

Let's write down Fibonacci!

K = 0 1 1 2 3 5 ...

And another series that is derived from Fibo;

X = 1 4 7 12 30 $X_{5}$

Guessing $X_{5}$ is our task and both series are available to you (Fibonacci as an additional feature).

One unit that you feed with X will try to capture the relation of $X_{t}$ and $X_{t-1}$ only;

$X_{t}$ = $X_{t-1}$ + $X_{t-2}$

However, an additional unit that you insert&feed with K will not only try to capture $K_{t}$ and $K_{t-1}$ relation but also the relation between $K_{t}$ and $X_{t}$.

$K_{t}$ = $K_{t-1}$ + $2*$$X_{t}$ + $1$

In the example above, there is a clear correlation between $K_{t}$ and $X_{t}$ (which is not always the case) and it will support the network to capture the sequential relation. Even the CORRELATION is not crystal clear, almost every additional feature data correlates with the other features and will support the network to grab the relation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.