In the problems of NLP and sequence modeling Transformer architectures based on self-attention mechanism https://arxiv.org/abs/1706.03762 have achieved impressive results and now are the first choices in this sort of problems.

However, most of the architectures, which appear in the literature have a lot of parameters and are aimed at solving rather complicated tasks of language modeling (https://arxiv.org/abs/1810.04805, https://arxiv.org/abs/2005.14165). These models have large amount of parameters and are computationally expensive.

There exist multiple approaches to reduce the computational complexity of these model, like knowledge distillation https://medium.com/pytorch/bert-distillation-with-catalyst-c6f30c985854?sk=1a28469ac8c0e6e6ad35bd26dfd95dd9 or multiple approaches to deal with $O(n^2)$ computational complexity of the self-attention (https://arxiv.org/abs/2006.04768, https://arxiv.org/abs/2009.14794).

However, this models are still aimed at language modelling and require quite a lot of params.

I wonder, whether there successful application os transformers with very small number of parameters (1k-10k) in the signal processing applications, where inference has to be performed in a very fast way, hence heavy and computationally expensive models are not allowed. So far, the common approaches are CNN or RNN architectures, but I wonder, whether there are some results, where lightweight transformers have achieved SOTA results for these extremely small models?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.