1
$\begingroup$

I am aware that the attention mechanism can be used to deal with long sequences, where problems related to gradient vanishing and, more generally, representing effectively the whole sequence arise.

However, I was wondering if attention, applied either to seq2seq RNN/GRU/LSTM or via Transformers, can contribute to improving the overall performance (as well as giving some sort of interpretability through the attention weights?) in the case of relatively short sequences (let's say around 20-30 elements each).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

They shouldn't have any issues with short sequences, as short dependencies are easier to learn. The only difficult cases are long dependencies which is where most of the research is geared at. However, this is assuming that by "short sequence" you mean a sequence of text that is fully contained within itself, i.e. there is no cross-sequence dependencies.

For example, if you have a really long paragraph that doesn't fit in a transformer model, you would have to break that paragraph into many "short sequences", but each of these sequence may have a dependency that depends on another sequence, i.e. cross-sequence dependencies. For these cross-sequence dependencies, any model with recurrence should do better than ones without (e.g. RNN, LSTM, Transformer-XL).

If each short sequence is self-contained, then all of the models should perform pretty well.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you very much for the explanation. I am not really dealing with NLP, but with sequences which have a reasonably similar representation. All those sequences are self-contained. My idea was to exploit attention (I was using plain LSTMs, with quite good performance) to leverage the fact that I know that some elements of my input sequence are more relevant to predict certain elements of the output sequence (the problem is encoded as a seq2seq one). So, in this context, attention, even if I am dealing with short sequences, could be helpful? $\endgroup$
    – nsacco
    Dec 19, 2020 at 20:42
  • 1
    $\begingroup$ Yes, if your task parallels a seq2seq task, then it is very likely Transformers will do better than LSTM on ideal conditions. But if you have a small transformer model, small dataset, or small hardware, transformers will likely to perform worse than LSTM. The reason transformers push state of the art is because companies can train extremely huge models on extremely huge datasets very quickly. But if you are training a model from scratch and you do not have powerful hardware and millions of data points, I think LSTM would be more practical. $\endgroup$
    – user3667125
    Dec 19, 2020 at 22:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .