In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time. What if my sequence is larger than 128? How to interpret this? Can anyone please explain me? Thank in advance.
In Keras, what you specify is the hidden layer size. So :
gives you a Keras layer representing a LSTM with a hidden layer size of 128.
As you said :
From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time
So if you picture your RNN for one time step, it will look like this :
And if you unfold it in time, it look like this :
You are not limited in your sequence size, this is one of the feature of RNN : since you input your sequence element by element, the size of the sequence can be variable.
That number, 128, represent just the size of the hidden layer of your LSTM. You can see the hidden layer of the LSTM as the memory of the RNN.
Of course the goal is not for the LSTM to remember everything of the sequence, just link between elements. That's why the size of the hidden layer can be smaller than the size of your sequence.
From this blog :
The larger the network, the more powerful, but it’s also easier to overfit. Don’t want to try to learn a million parameters from 10,000 examples – parameters > examples = trouble.
So the consequence of reducing the size of hidden state of LSTM is that it will be simpler. Might not be able to get the links between the element of the sequence. But if you put a too big size, your network will overfit ! And you absolutely don't want that.
Another really good blog on LSTM : this link
Although this question has been answered I'd add a couple of remarks towards general neural networks design
As you know very NN has three types of layers: input, hidden, and output. Once this network is initialized, you can iteratively tune the configuration during training.
To optimize the network configuration we can use pruning.
Pruning describes a set of techniques to trim network size (by nodes not layers) to improve computational performance and sometimes resolution performance. The gist of these techniques is removing nodes from the network during training by identifying those nodes which, if removed from the network, would not noticeably affect network performance (i.e., resolution of the data). (Even without using a formal pruning technique, you can get a rough idea of which nodes are not important by looking at your weight matrix after training; look weights very close to zero--it's the nodes on either end of those weights that are often removed during pruning.)