Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other problems)?

  • $\begingroup$ Image generation with transformers instead of GANs is a thing. $\endgroup$
    – thepacker
    Jun 25 '20 at 7:51
  • $\begingroup$ do you have a paper or a reference? $\endgroup$
    – Schach21
    Jun 25 '20 at 17:02
  • 1
    $\begingroup$ cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf see also the completions section at their website. openai.com/blog/image-gpt $\endgroup$
    – thepacker
    Jun 25 '20 at 17:37
  • $\begingroup$ I've recently start seeing transformer-based models being used for traffic forecasting. Here is an example: arxiv.org/abs/2001.02908 $\endgroup$
    – razvanc92
    Aug 3 '20 at 8:27
  • $\begingroup$ @razvanc92 Although this is an old question, you can provide a formal answer below. Comments are meant to be temporary. $\endgroup$
    – nbro
    Nov 30 '21 at 15:35

When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.

OpenAI did some related research, as the followings.

Generative Modeling with Sparse Transformers: In the paper, transformers with sparse attention are applied to image and waveform.

ImageGPT: A large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. (Abstract from the blog)

  • $\begingroup$ You may also want to mention the vision transformer. $\endgroup$
    – nbro
    Nov 30 '21 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.