Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice.

For example, in Keras:

Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);

However, with lots of sequences to train, should I add more LSTM layers? because increasing MAX_SEQ_LEN is not the way as it doesn't help make the network better since the extra number of hidden states isn't useful any more.

I'm considering increasing number of LSTM layers, but how many are enough?

For example, 3 of them:

Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
Lstm2 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
Lstm3 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);

From my personal experience, the units hyperparam in LSTM is not necessary to be the same as max sequence length. Add more units to have the loss curve dive faster.

And about the number of LSTM layers, trying out a single LSTM layer is a good start point, the model trains better with more LSTM layers.

For example, MAX_SEQ_LEN=10, in Keras:

Lstm1 = LSTM(units=30, return_sequences=True); #Time sequence, to feed to next layer
Lstm2 = LSTM(units=20, return_sequences=True); #Time sequence, to feed to next layer
Lstm3 = LSTM(units=10, return_sequences=False);

Output = Lstm3(Lstm2(Lstm1));

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.