I have a problem with a subset of my data which is as follows:

I can train a model (doesn't matter what, xgboost, BERT, etc., it is a text classification problem), on my data and get a decent performance. Now it turns out that there is a specific subset I can access where the performance is explicitly worse. On the contrary I can obviously exclude this data and get a better performance.

Now, I know that the final input distribution will obviously contain samples which would fall into this "bad" subset, so simply excluding it in the training will probably not suffice. A model trained on data which doesn't follow the "real-world" distribution will obviously be worse in the "real-world".

Is there any way I can deal with this knowledge? Maybe weighting the subset in the training? Or preprocessing in a way? Or is there a specific technical term for this so I can do research myself?

I'll gladly provide more information if needed, but would also like to keep the question as general as possible.

  • $\begingroup$ I have learned that these are called "hard samples" by some. There seems to be no consensus to deal with those, but I am open to every suggestion and discussion. $\endgroup$ – N. Kiefer Jul 3 '20 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.