I have a training dataset of 80 text documents with an average number of characters in each document of 25000 and 210 unique tags.

How can I perform multi-class text classification with such a small dataset, without using the pre-trained model? If it cannot be done without a pre-trained model, then which pre-trained model should I use?


1 Answer 1


For pretrained models in NLP, look at BERT and RoBERTa. If you can find a language model trained on your data's superset on Huggingface, then, use that pretrained model.

In order to multiclass classification, since your data is less, look at augmentations in NLP (most notably, backtranslation amongst others). Use focal loss (to handle class imbalance).

Since, you are going to finetune use small learning, 1e-5. But, you will be adding your own layers also, so keep 1e-5 for the pretrained model and 1e-3 for the new layer you put.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .