I am trying to make a text score prediction network, and my dataset have 500 samples only. I know there is a public dataset called the ASAP Dataset. I have tested my model (word embedding layer --> LSTM Layer --> FC Layer) on this dataset and it performed as expected. The public dataset have 13000 data samples while my private one has only 500. When I trained the network on my private dataset, it performed very poorly, and started overfitting from the first epoch. I have tried reducing my model size to the minimum and still no improvement. I have also added dropout and l2 regularization ans still nothign works. Is there any suggestion that could help like a differnet model or something. I am thinking methods like cropping the text to generate more data or some other methods like a Siamese network approach will help. Will they help? For your refernce, this is my code. https://github.com/Clement-Hui/EssayGrading Thank you very much for your help. I would be glad to add a bounty if the answer really helps me. Thanks.

  • 1
    $\begingroup$ you can use transfer learning for a small dataset. Look in the fastai library they have used AWD-LSTM with ULMFIT for transfer learning. $\endgroup$ – aman5319 Nov 18 '19 at 13:19
  • $\begingroup$ I have tried transfer learning and got bad results ( 0.3 kappa). do I have to lock specific layers? i locked the weights of the lstm layer only. $\endgroup$ – Clement Hui Nov 18 '19 at 14:27
  • 1
    $\begingroup$ How did you do the transfer learning?? what approach did you took mention that. $\endgroup$ – aman5319 Nov 18 '19 at 14:42
  • $\begingroup$ I used the ASAP dataset to pretrain the model and stops the model when it starts to overfit. Then I used the model to train with my private dataset, locking the weights of the LSTM layer, and only allow thw wmbedding layer and the output FC layer to train. $\endgroup$ – Clement Hui Nov 18 '19 at 15:08
  • 1
    $\begingroup$ The Method which you described is the wrong way of doing transfer learning use fast.ai library for that. $\endgroup$ – aman5319 Nov 18 '19 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.