In the problems of NLP and sequence modeling Transformer architectures based on self-attention mechanism https://arxiv.org/abs/1706.03762 have achieved impressive results and now are the first choices in this sort of problems.
However, most of the architectures, which appear in the literature have a lot of parameters and are aimed at solving rather complicated tasks of language modeling (https://arxiv.org/abs/1810.04805, https://arxiv.org/abs/2005.14165). These models have large amount of parameters and are computationally expensive.
There exist multiple approaches to reduce the computational complexity of these model, like knowledge distillation https://medium.com/pytorch/bert-distillation-with-catalyst-c6f30c985854?sk=1a28469ac8c0e6e6ad35bd26dfd95dd9 or multiple approaches to deal with $O(n^2)$ computational complexity of the self-attention (https://arxiv.org/abs/2006.04768, https://arxiv.org/abs/2009.14794).
However, this models are still aimed at language modelling and require quite a lot of params.
I wonder, whether there successful application os transformers with very small number of parameters (1k-10k) in the signal processing applications, where inference has to be performed in a very fast way, hence heavy and computationally expensive models are not allowed. So far, the common approaches are CNN or RNN architectures, but I wonder, whether there are some results, where lightweight transformers have achieved SOTA results for these extremely small models?