I would like to build an LSTM to predict the correct words order given a sentence. My dataset is composed of sentences, where each sentence has a variable number of words (each word is embedded). The dataset then is an array of matrices, where each matrix is an array of embedded words.

Now, I'm looking to implement it with Keras but I'm not sure how to fit the necessary parameters wanted by the LSTM layer in Keras, like timesteps and batch_size.

Reading on the web, I notice that timesteps is the length of the sequence, so in my case I believe that corresponds to the length of the sentence. But I want to train my LSTM with one sentence at a time, so would the batch_size be 1?

  • $\begingroup$ I’m closing this question because general programming questions (including how to use an API or library) are off-topic. See our on-topic page: https://ai.stackexchange.com/help/on-topic. $\endgroup$ – nbro May 2 '20 at 22:14

As you read, in keras the input dimensions for a LSTM layer are: (batch_size, timesteps, input_dim). Where:

  • batch_size: number of samples (sentences in your case) to compute the loss before running the gradient descent. So number of sentences to train on before compute loss and optimize your model.
  • timesteps: length of the sequence, in your case length of the sentence
  • input_dim: the features of your data (the words)

The nice thing about Keras is that you can train with an specific batch size, say batch_size=16, that: helps the model convergence (because it averages the loss of the 16 sentences prediction) and boosts the speed (since the weights are updated only once every 16 sentences).

But then you can infer (or do the predictions) with batch_size=1, meaning, one sentence at a time.

So, if you want to train, specifically with one sentence at a time: batch_size=1. But if you want to take the advantage of using mini-batch then use batch_size=16 or higher for training and batch_size=1 for inference.

BONUS: Keras allows for dynamic batch size change in the models. When a dimension is dynamic in Keras has the value of None, that is why when you do model.summary() you can see: (batch_size, timesteps, input_dim) = (None, 40, 100). This None allows for different batch_size in training and inference

  • $\begingroup$ Thank you very much for your answers: I'll try to implement it. $\endgroup$ – pairon Feb 11 '20 at 11:43
  • $\begingroup$ @JVGC is it right pass to the model.fit()a 3D dataset (list of matrix, each matrix is a list of words)? And if it's right, which should be the dimension of the labels? $\endgroup$ – pairon Feb 11 '20 at 20:54
  • $\begingroup$ @pairon, as long as the input data have the dimension of the model inputs, it is right. Normally .fit() is used for when data can be loaded into memory and there is no data augmentation. On the other hand, .fit_generator() is used for when data do not fit in memory. But the .fit() input only needs to mach the model inputs. $\endgroup$ – JVGD Feb 12 '20 at 8:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.