I am working on a project where I have to first classify the Subjects of the given question and then the respective Chapter and then the sub-topic. In a nutshell, I have to predict the Subject, Grade and Chapter Name of a Given text. I have trained my Subject Classification model with BERT last 40% trainable + Focal Loss + Class Weights + 4 output classes and got around
0.91 f-1 macro as there is imbalance in my dataset. The next problem is to classify the Chapter name of the question.
So my problems have tokens like
fe(iron), kj (kilo joule), ligand, pascal, ppm which are subjject specific and I can not afford to discard any info. Please tell me if
- Can I use FastText, GloVe or some other embeddings because I don't think they'll able to capture these things? Or should I use my own Embeddings?
- Which one of the above models in your opinions do you think will give me good results? I don't think if BERT is capable of capturing the subjject wise info or should I train my Attention/Self-Attention/Transformer model from scratch?
I am using a dataset of 120000 text questions. Please suggest some strategy.