4
$\begingroup$

What is the point of having multiple LSTM units in a single layer?

Surely if we have a single unit it should be able to capture (remember) all the data anyway and using more units in the same layer would just make the other units learn exactly the same historical features?

I've even shown myself empirically that using multiple LSTMs in a single layer improves performance, but in my head it still doesn't make sense, because I don't see what is it that other units are learning that others aren't? Is this sort of similar to how we use multiple filters in a single CNN layer?

Improve this question
$\endgroup$
2
  • $\begingroup$ Your assumption "Surely if we have a single unit it should be able to capture (remember) all the data anyway" is likely incorrect. Why do you think a single unit would be sufficient? This is just an interpretation, but, if you have to keep track of multiple concepts at the same time step, then a single RNN unit might not be sufficient. $\endgroup$
    – nbro
    Commented May 15, 2019 at 22:15
  • $\begingroup$ that was my question i got here : ai.stackexchange.com/questions/12657/… : for RNN, multiple cells/units, whatever store the data from previous time stamps and use it for backpropagate. Just RNN cell store only weights abd biases, and LSTM stores more stuff what is in there $\endgroup$
    – user8426627
    Commented Jun 15, 2019 at 1:05

1 Answer 1

Reset to default
2
$\begingroup$

Let's write down Fibonacci!

K = 0 1 1 2 3 5 ...

And another series that is derived from Fibo;

X = 1 4 7 12 30 $X_{5}$

Guessing $X_{5}$ is our task and both series are available to you (Fibonacci as an additional feature).

One unit that you feed with X will try to capture the relation of $X_{t}$ and $X_{t-1}$ only;

$X_{t}$ = $X_{t-1}$ + $X_{t-2}$

However, an additional unit that you insert&feed with K will not only try to capture $K_{t}$ and $K_{t-1}$ relation but also the relation between $K_{t}$ and $X_{t}$.

$K_{t}$ = $K_{t-1}$ + $2*$$X_{t}$ + $1$

In the example above, there is a clear correlation between $K_{t}$ and $X_{t}$ (which is not always the case) and it will support the network to capture the sequential relation. Even the CORRELATION is not crystal clear, almost every additional feature data correlates with the other features and will support the network to grab the relation.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .