1
$\begingroup$

In the problems of NLP and sequence modeling Transformer architectures based on self-attention mechanism https://arxiv.org/abs/1706.03762 have achieved impressive results and now are the first choices in this sort of problems.

However, most of the architectures, which appear in the literature have a lot of parameters and are aimed at solving rather complicated tasks of language modeling (https://arxiv.org/abs/1810.04805, https://arxiv.org/abs/2005.14165). These models have large amount of parameters and are computationally expensive.

There exist multiple approaches to reduce the computational complexity of these model, like knowledge distillation https://medium.com/pytorch/bert-distillation-with-catalyst-c6f30c985854?sk=1a28469ac8c0e6e6ad35bd26dfd95dd9 or multiple approaches to deal with $O(n^2)$ computational complexity of the self-attention (https://arxiv.org/abs/2006.04768, https://arxiv.org/abs/2009.14794).

However, this models are still aimed at language modelling and require quite a lot of params.

I wonder, whether there successful application os transformers with very small number of parameters (1k-10k) in the signal processing applications, where inference has to be performed in a very fast way, hence heavy and computationally expensive models are not allowed. So far, the common approaches are CNN or RNN architectures, but I wonder, whether there are some results, where lightweight transformers have achieved SOTA results for these extremely small models?

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.