I have a dataset where each of the training instances is different in the length and the data is sequential. So, I design an LSTM but I am thinking about how to train the LSTM. In fixed-length data, we just keep all of the input in an array and pass it to the network, but here the case is different. I can not store varying length data in an array and I do not want to use padding to make it fixed length. Now, should I train the LSTM where each training instance are varying in length?

  • $\begingroup$ Are you predicting a sequence e.g. like in machine translation? $\endgroup$ Jul 30 '20 at 8:19
  • $\begingroup$ I am performing prediction but not like the machine translation $\endgroup$ Jul 30 '20 at 8:22

You could sequentially pass in each element of your sequential data and save the hidden and cell states in a separate buffer. In a typical LSTM implementation, you input the entire sequence and the hidden and cell states are propagated internally. In the end, the final hidden and cell states returned as the output. This works if your input is all the same length. Instead, you can handle sequentially giving the next element to the LSTM as well as the hidden and cell state yourself. To keep it efficient you can batch your inputs by the batch dimension (batch_first=True in the pytorch LSTM implementation).

For example, propose you have 5 sequences of length 5, 4, 3, 2, and 1. You initialize your hidden and cell state buffers for each of the sequences and pass the first batch containing the first element of all 5 sequences. You save the output hidden and cell states in the buffers for each sequence. Next, you input the batch of the second element of the 4 sequences with length > 1, and save the states in the respective sequence buffers and so and so forth until you exhaust the sequence of greatest length.

  • $\begingroup$ What if all of the sequences are different length, so I can not batch the same length sequence together? $\endgroup$ Jul 30 '20 at 9:11
  • $\begingroup$ You don't need to batch the sequences, you just need to batch the elements of the sequences. Batch all the first elements of your sequences into one vector (or matrix if your elements are multidimensional), all the second elements in another etc. $\endgroup$ Jul 30 '20 at 9:15
  • $\begingroup$ I do not want to pass the first batch containing the first element of all 5 sequences like this, I am also curious about why one should pass input like this? $\endgroup$ Jul 30 '20 at 9:28
  • $\begingroup$ Because this avoids having to pad your input to efficiently process variable length sequences. AFAIK the LSTM implementations in all typical python packages receive input of size Batch x Sequence x Hidden (or a permutation). Since your sequence dimension is not the same for every sequence, you either need to pad your sequences. Or as suggested in the answer, you can change the sequence dimension to 1, but need to handle the hidden and cell states of the LSTM yourself. $\endgroup$ Jul 30 '20 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.