How to train an LSTM with varying length input?

Question

I have a dataset where each of the training instances is different in the length and the data is sequential. So, I design an LSTM but I am thinking about how to train the LSTM. In fixed-length data, we just keep all of the input in an array and pass it to the network, but here the case is different. I can not store varying length data in an array and I do not want to use padding to make it fixed length. Now, should I train the LSTM where each training instance are varying in length?

Answer 1

You could sequentially pass in each element of your sequential data and save the hidden and cell states in a separate buffer. In a typical LSTM implementation, you input the entire sequence and the hidden and cell states are propagated internally. In the end, the final hidden and cell states returned as the output. This works if your input is all the same length. Instead, you can handle sequentially giving the next element to the LSTM as well as the hidden and cell state yourself. To keep it efficient you can batch your inputs by the batch dimension (batch_first=True in the pytorch LSTM implementation).

For example, propose you have 5 sequences of length 5, 4, 3, 2, and 1. You initialize your hidden and cell state buffers for each of the sequences and pass the first batch containing the first element of all 5 sequences. You save the output hidden and cell states in the buffers for each sequence. Next, you input the batch of the second element of the 4 sequences with length > 1, and save the states in the respective sequence buffers and so and so forth until you exhaust the sequence of greatest length.

Answer 2

This question might be deep learning framework specific, so my answer is intended for Keras.

In keras, an LSTM cell accept 3D tensor, which is (num of samples, num of timesteps, num of features). The question is about varying num of timesteps.

My naive approach will be like this:

1. Define model architecture in a function that accept num of timestep as parameter that returning model object
2. Prepare next sample
3. Instantiate a model with num of timestep based on current sample
4. Load the latest weights if exist
5. Train one epoch
6. Save the latest weights
7. Back to number 2

So, the code will be looks like this:

from keras.models import Model
from keras.layers import LSTM, Input, Dense
import numpy as np

def my_robust_lstm_model(num_of_timesteps=512, num_of_features=1):
    input_layer = Input(shape=(num_of_timesteps, num_of_features))
    hidden_layer = LSTM(64)(input_layer)
    output_layer = Dense(1)(hidden_layer)  # Example output layer
    model = Model(inputs=input_layer, outputs=output_layer)
    return model

def array_generator(my_list_pickle_path):
    # Implement how one sample X_train, y_train looks like that has varying length
    # Example implementation: (Replace this with actual data generation logic)
    while True:
        X_train = np.random.random((1, 10, 1))  # Dummy data with varying lengths
        y_train = np.random.random((1, 1))
        yield X_train, y_train

generator = array_generator('path/to/pickle/file')
EPOCHS = 10

for _ in range(EPOCHS):
    current_X, current_y = next(generator)
    num_of_timesteps = current_X.shape[1]
    num_of_features = current_X.shape[2]
    current_model = my_robust_lstm_model(num_of_timesteps=num_of_timesteps, num_of_features=num_of_features)
    
    try:
        current_model.load_weights('latest.weights.h5')
    except Exception as e:
        print(f'Weights don\'t exist or cannot be loaded: {e}, skipping')
    
    current_model.compile(optimizer='adam', loss='mse')  # Example compile parameters
    current_model.fit(current_X, current_y, batch_size=1, epochs=1)
    current_model.save_weights('latest.weights.h5')