0
$\begingroup$

I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be just 0 and 1, determining if the word level translation is good or bad on both the source and target sides.

To begin, I used XLMRoberta, as I thought it would be best suited for my problem. First, I just trained normally using nothing fancy, but the model overfits after just 1-2 epochs, as I have very little data to fine-tune on (approx 7k).

I decided to freeze the BERT layers and just train the classifier weights, but it performed worse.

I thought of adding a more dense network on top of BERT, but I am not sure if it would work well or not.

One more thought that occurred to me was data augmentation, where I increased the size of my data by multiple factors, but that performed badly as well. (Also, I am not sure what should be the proper number to increase the data size with augmented data.)

Can you please suggest which approach could be suitable here and if I am doing something wrong? Shall I just use all the layers for my data or freezing is actually a good option? Or you suspect I am ruining somewhere in the code and this is not what is expected.

Improve this question
$\endgroup$
3
  • $\begingroup$ What do you mean by "I just trained normally using nothing fancy"? Do you mean you're using supervised learning with a labelled dataset? Or are you using self-supervised learning? Or maybe semi-supervised learning, as you wrote at the beginning of your post? It's not clear how much data you're using and for what. Are you just fine-tuning this "XLMRoberta", or training it from scratch? It seems that you also tried to fine-tune it, so I assume that you also trained it from scratch. $\endgroup$
    – nbro ♦
    Dec 10 '21 at 9:10
  • $\begingroup$ In both cases, you need to explain 1. how much training, validation and test data do you have 1.1. when you train from scratch, 1.2. when you fine-tune. $\endgroup$
    – nbro ♦
    Dec 10 '21 at 9:10
  • $\begingroup$ Thanks @nbro , sorry for the less information. By nothing fancy, i meant i fine tuned the xlmroberta for the problem without augmentation, just the normal training dataset(7k) and validated for 1k dataset. I havent used semi supervised learning as of now, i am just doing a typical supervised fashion. I cannot train the model from scratch as i have a little dataset , so i was trying ti improve my finetuning with augmented dataset, which did not helped. $\endgroup$
    – Tarun Bhatia
    Dec 10 '21 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.