In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time. What if my sequence is larger than 128? How to interpret this? Can anyone please explain me? Thank in advance.


In Keras, what you specify is the hidden layer size. So :


gives you a Keras layer representing a LSTM with a hidden layer size of 128.

As you said :

From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time

So if you picture your RNN for one time step, it will look like this :

enter image description here

And if you unfold it in time, it look like this :

enter image description here

You are not limited in your sequence size, this is one of the feature of RNN : since you input your sequence element by element, the size of the sequence can be variable.

That number, 128, represent just the size of the hidden layer of your LSTM. You can see the hidden layer of the LSTM as the memory of the RNN.

Of course the goal is not for the LSTM to remember everything of the sequence, just link between elements. That's why the size of the hidden layer can be smaller than the size of your sequence.

Sources :


From this blog :

The larger the network, the more powerful, but it’s also easier to overfit. Don’t want to try to learn a million parameters from 10,000 examples – parameters > examples = trouble.

So the consequence of reducing the size of hidden state of LSTM is that it will be simpler. Might not be able to get the links between the element of the sequence. But if you put a too big size, your network will overfit ! And you absolutely don't want that.

Another really good blog on LSTM : this link

  • $\begingroup$ Thank you for your answer. What if I lower the size of the hidden layer? say LSTM(64). What will be the consequences? Does it means lstm model has lower memory to capture long-term dependencies? $\endgroup$ Oct 15 '18 at 13:47
  • $\begingroup$ I edited the answer :) $\endgroup$
    – Astariul
    Oct 15 '18 at 23:32

Although this question has been answered I'd add a couple of remarks towards general neural networks design

As you know very NN has three types of layers: input, hidden, and output. Once this network is initialized, you can iteratively tune the configuration during training.

To optimize the network configuration we can use pruning.

Pruning describes a set of techniques to trim network size (by nodes not layers) to improve computational performance and sometimes resolution performance. The gist of these techniques is removing nodes from the network during training by identifying those nodes which, if removed from the network, would not noticeably affect network performance (i.e., resolution of the data). (Even without using a formal pruning technique, you can get a rough idea of which nodes are not important by looking at your weight matrix after training; look weights very close to zero--it's the nodes on either end of those weights that are often removed during pruning.)

You can find more here: https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.