I'm working on a comparative study using some models in a sentiment analysis task: MLPs and LSTMs with and without the use of word embeddings (GloVe and Word2Vec) and two Transformer-based models (BERT and XLNet).

I was surprised by the fact that the MLPs had a performance superior even to the models that used BERT and XLNet (the values below are the mean MCC values extracted from the 10-fold CV):

enter image description here

enter image description here

I defined my MLP models this way:

def MLP1x128model(input_shape):
  X_indices = Input(input_shape)
  embeddings = Embed_Layer(X_indices)
  dropout = Dropout(0.5)(embeddings)
  Flatten_Layer = Flatten()
  flatten = Flatten_Layer(dropout)
  dense1 = Dense(128,activation='relu')(flatten)
  predict = Dense(3,activation='softmax')(dense1)

  model = Model(inputs=X_indices, outputs=predict)

  return model

What could have caused this behavior, since they are much simpler models and are far from the current state-of-the-art?

  • $\begingroup$ "Occam's Razor" - Simple stuff work, you don't need complicated things. SOTA models do not mean they perform the best in every task. Transformer models are generally designed for slightly more complex NLP tasks and they perform with huge data well. For simple tasks, they may at times perform really badly. With respect to NLP it can either be due to the nature of the data (language) or just not enough data. Remember to cut a piece of cake a Ninja sword does the job bad compared to a simple small knife. $\endgroup$ Jul 11 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.