I'm working on a comparative study using some models in a sentiment analysis task: MLPs and LSTMs with and without the use of word embeddings (GloVe and Word2Vec) and two Transformer-based models (BERT and XLNet).
I was surprised by the fact that the MLPs had a performance superior even to the models that used BERT and XLNet (the values below are the mean MCC values extracted from the 10-fold CV):
I defined my MLP models this way:
def MLP1x128model(input_shape): X_indices = Input(input_shape) embeddings = Embed_Layer(X_indices) dropout = Dropout(0.5)(embeddings) Flatten_Layer = Flatten() flatten = Flatten_Layer(dropout) dense1 = Dense(128,activation='relu')(flatten) predict = Dense(3,activation='softmax')(dense1) model = Model(inputs=X_indices, outputs=predict) return model
What could have caused this behavior, since they are much simpler models and are far from the current state-of-the-art?