What is the point of having multiple LSTM units in a single layer?

Surely if we have a single unit it should be able to capture (remember) all the data anyway and using more units in the same layer would just make the other units learn exactly the same historical features?

I've even shown myself empirically that using multiple LSTMs in a single layer improves performance, but in my head it still doesn't make sense, because I don't see what is it that other units are learning that others aren't? Is this sort of similar to how we use multiple filters in a single CNN layer?

  • $\begingroup$ Your assumption "Surely if we have a single unit it should be able to capture (remember) all the data anyway" is likely incorrect. Why do you think a single unit would be sufficient? This is just an interpretation, but, if you have to keep track of multiple concepts at the same time step, then a single RNN unit might not be sufficient. $\endgroup$ – nbro May 15 '19 at 22:15
  • $\begingroup$ that was my question i got here : ai.stackexchange.com/questions/12657/… : for RNN, multiple cells/units, whatever store the data from previous time stamps and use it for backpropagate. Just RNN cell store only weights abd biases, and LSTM stores more stuff what is in there $\endgroup$ – user8426627 Jun 15 '19 at 1:05

Let's write down Fibonacci!

K = 0 1 1 2 3 5 ...

And another series that is derived from Fibo;

X = 1 4 7 12 30 $X_{5}$

Guessing $X_{5}$ is our task and both series are available to you (Fibonacci as an additional feature).

One unit that you feed with X will try to capture the relation of $X_{t}$ and $X_{t-1}$ only;

$X_{t}$ = $X_{t-1}$ + $X_{t-2}$

However, an additional unit that you insert&feed with K will not only try to capture $K_{t}$ and $K_{t-1}$ relation but also the relation between $K_{t}$ and $X_{t}$.

$K_{t}$ = $K_{t-1}$ + $2*$$X_{t}$ + $1$

In the example above, there is a clear correlation between $K_{t}$ and $X_{t}$ (which is not always the case) and it will support the network to capture the sequential relation. Even the CORRELATION is not crystal clear, almost every additional feature data correlates with the other features and will support the network to grab the relation.


The purpose of multiple LSTM units (or cell) in an artificial network layer is to represent the time dimension discretely as a sequence of samples in that layer. This time sequence is quite different than the sequence of examples processed in a batch. In the temporal sequence, there is a variance over time that We want to preserve during processing. A layer performs a stage in that processing. If any stage compresses the time domain by reducing the bit width of the signal path too quickly in the circuit that forms as training continues, the relationship between phenomena being learned and time is lost.

In artificial networks in general, the narrowing of a layer with respect to the previous layer (in terms of the number of cells per layer) is a forced reduction in data bandwidth per example. Because of the convergence approach require to train artificial networks, this compression is not simply the dropping of some arbitrarily selected data, but the abstraction of the data so that it fits through the narrow. The compression that occurs is not lossless, but the nature of the loss is controlled by the convergence, driven by the loss function, although that's not why it is called a loss function. Ideally, the valuable aspects of the information in the data are retained and what information is lost during the compression is not relevant to the objective defined by the loss function's descent toward zero. This is the nature of feature extraction.

Such reductions are plausibly helpful in LSTM or GRU cell layers, just as they are in basic Perceptron layers, but not necessarily so. In some cases narrowing the signal path through a layer by reducing the number of cells is counterproductive to the objective at that depth into the network. Dropping down to one cell width for a layer is far too drastic (unless your value trend is $v = at + b$) and will lose everything in the data representing variance over time.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.