I have recently come across transformers, I am new to Deep Learning. I have seen a paper using CNN and BiLSTM on top of a transformer, the paper uses a transformer(XLM-R) for sentiment analysis in code-mixed domain. But many of the blogs only use a normal feed formal network on top of the transformer.

I am trying to use transformers for sentiment analysis, short text classification.

Is it overkill to use models like CNN and BiLSTM on top of the transformer considering the size of the data it is trained on and its complexity?

  • $\begingroup$ Hello. Welcome to AI SE. It would be better if you describe the problem you're trying to solve with the transformer and LSTM and what that paper is using them for. $\endgroup$ – nbro Jan 25 at 3:53
  • $\begingroup$ The paper is using transformers for sentiment analysis on code mixed domain $\endgroup$ – learner Jan 25 at 4:34
  • $\begingroup$ It may be useful to also know what those blog posts that you mention use the LSTM + transformer for. Do they use it for the same task? Maybe provide a link to one such blog post. $\endgroup$ – nbro Jan 25 at 12:42
  • $\begingroup$ medium.com/atheros/… $\endgroup$ – learner Jan 26 at 4:54
  • $\begingroup$ The hugging face implementation of transformers for sequence classification uses a feed-forward layer on top of the transformer $\endgroup$ – learner Jan 26 at 4:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.