Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other problems)?

  • $\begingroup$ Image generation with transformers instead of GANs is a thing. $\endgroup$
    – thepacker
    Jun 25, 2020 at 7:51
  • $\begingroup$ do you have a paper or a reference? $\endgroup$
    – Schach21
    Jun 25, 2020 at 17:02
  • 1
    $\begingroup$ cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf see also the completions section at their website. openai.com/blog/image-gpt $\endgroup$
    – thepacker
    Jun 25, 2020 at 17:37
  • $\begingroup$ I've recently start seeing transformer-based models being used for traffic forecasting. Here is an example: arxiv.org/abs/2001.02908 $\endgroup$
    – razvanc92
    Aug 3, 2020 at 8:27
  • $\begingroup$ @razvanc92 Although this is an old question, you can provide a formal answer below. Comments are meant to be temporary. $\endgroup$
    – nbro
    Nov 30, 2021 at 15:35

1 Answer 1


When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.

OpenAI did some related research, as the followings.

Generative Modeling with Sparse Transformers: In the paper, transformers with sparse attention are applied to image and waveform.

ImageGPT: A large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. (Abstract from the blog)

  • $\begingroup$ You may also want to mention the vision transformer. $\endgroup$
    – nbro
    Nov 30, 2021 at 15:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .