Data preparation for NLP model

Question

I have data from our ticketing system. Currently using OpenNLP to create different models.

For simplicity I have a 10k ticket's text as category final queue of the ticket.

My questions:

• Is it important for the model to have data similarly distributed?
  • eg. for 3 category imagine that 6k items is for 1.cat, 3k is for 2.cat and the rest 1k for 3.cat.
  • Will this affect the final classification?
• Is it wise to remove a constants from the evaluated text? eg. "Good day", "Best regards" and others?
  • Should I already remove such constants from data set for training model or just remove it from text for classification?

Answer 1

1. You should use a metric like precision or recall (whatever is more relevant to your use case) that is robust against dataset bias. You may also consider oversampling the minority classes during training.

2. If you ask me, the easiest way to do this would be to convert whole text to embeddings using a pre-trained language model. Then train a small classification MLP on top of those text embeddings.

Answer 2

1. depends, that imbalance will induce a prior over your model: think about the case of malware detection, where 99% of the files are ok, and 1% are malware... in this case if your model always predicts "non malware", it has a 99% accuracy, due to the fact that you have a strong prior

2. depends, if you think that such pieces are irrelevant for the classification, you can definitely do that