# Do transformers have success in other domains different than NLP?

Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other problems)?

• Image generation with transformers instead of GANs is a thing. Jun 25 '20 at 7:51
• do you have a paper or a reference? Jun 25 '20 at 17:02
• cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf see also the completions section at their website. openai.com/blog/image-gpt Jun 25 '20 at 17:37
• I've recently start seeing transformer-based models being used for traffic forecasting. Here is an example: arxiv.org/abs/2001.02908 Aug 3 '20 at 8:27
• @razvanc92 Although this is an old question, you can provide a formal answer below. Comments are meant to be temporary.
– nbro
Nov 30 '21 at 15:35

When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $$n^2$$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.