In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time t on each of the earlier hidden states (from 1 to t - 1). This is described here: enter image description here (image from https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b)

However, there is something I don't understand. This second neural network will output t values. So there will be t neurons on the last layer. What if my input sentence (in a seq2seq network for example) has only t - 2 words. Or t + 2 words. Then I can't use that same second neural network since its architecture is fixed. How is it dealt?

EDIT: is it that the second neural network takes as input one encoder hidden state and one decoder hidden state, and we get t scores by doing t forward passes (one per each input hidden state, all with the current decoder hidden state)? And then we softmax all those t outputs?


2 Answers 2


You don't need padding for attention with variable-length inputs. Looking at the formulation in the article: Attention description from article: https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b

A learned model ($\mathbf{a}(\cdot)$) encodes the hidden state of each input token directly. This means that if you have a longer/shorter input, you would just run more forward-passes.

During training, $\mathbf{a}(\cdot)$ learns to give higher values for more relevant tokens. It only depends on the hidden-dimensions of $h_j$ and $s_{t-1}$. You might still use masking and padding, this generally is more of a efficiency thing (because GPUs like static data sizes). In fact, you'll probably have to use masking to make sure that your decoder doesn't attend to masking tokens.

  • 1
    $\begingroup$ Oh I see, so we just do as many forward passes as potential hidden states we want to attend to, each forward pass giving a score and then we apply a softmax! $\endgroup$ Commented Feb 6 at 7:18
  • $\begingroup$ Yep, that's correct $\endgroup$ Commented Feb 6 at 18:07

Two concepts are critical here: masking and padding.

From Tensorflow:

Masking is a way to tell sequence-processing layers that certain timesteps in an input are missing, and thus should be skipped when processing the data.

Padding is a special form of masking where the masked steps are at the start or the end of a sequence. Padding comes from the need to encode sequence data into contiguous batches: in order to make all sequences in a batch fit a given standard length, it is necessary to pad or truncate some sequences.

See the article for more details.

Also see Introduction to Recurrent Neural Networks with Keras and TensorFlow by Aritra Roy Gosthipaty, Devjyoti Chakraborty, and Ritwik Raha. See the section entitled "A Caveat: Masking and Padding".

Prudhviraju Srivatsavaya explains the concepts in reference to Transformers in his article Attention Masks - Explanation.

  • $\begingroup$ So does that mean attention has a set length and if we have let's say more input tokens that its set length it would not be able to provide attention to any previous hidden states? $\endgroup$ Commented Feb 5 at 7:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .