I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. From my understanding when training such a model, you can encode the input in parallel, but the decoding is still sequential unless you're using teacher-forcing.

What makes the transformer faster than RNN in such a setting? Is there something that I am missing?

  • $\begingroup$ For those of you interested in the concept of teacher-forcing, have a look at the paper JT-VAE. Essentially, teacher-forcing is the trick of replacing the predictions with the actual ground-truth values, so that future predictions are based on correct past histories. $\endgroup$
    – nbro
    Oct 25 '19 at 1:19
  • $\begingroup$ @nbro Thank you for your comment. I am aware how teacher-forcing works but if you're training a model in such a way the model is not gonna learn how to recover from a mistake. On the other hand if scheduled sampling is used i think the model might perform better, but this will significantly increase the training time. $\endgroup$
    – razvanc92
    Oct 25 '19 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.