2
$\begingroup$

When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process?

My plan is to create a "small model" which can decide whether data used for training the final model are "good" or "bad".

For instance the small model will be made from "good" data example and "bad" data example. Later on all data which are going to be used to train a final big data model will be fist evaluated by the smaller one in order to increase overall quality of the data used in the final model.

$\endgroup$
4
  • $\begingroup$ What kind of NLP model are you building? I think the answer here will depend. If your finished product will be a sentiment analysis tool, probably a bad idea. If an autoregressive LLM, I don't know . . . But if you can say what it is, someone who knows the kind of model may be able to help $\endgroup$ Nov 18, 2023 at 8:21
  • $\begingroup$ @NeilSlater Building the automatic classification of the user help desk tickets (Queue, Category, Priority). Technology is OpenNLP from Apache. It works so far ok I still thinking how to make the model as good as possible. I believe the training data quality here is a key to gain a needed accuracy. Therefore filtering the "Too general tickets" might help. Indeed it will lead to the fact that the "general tickets" are going to be classified incorrectly but I think that is good. Because if the model do not provide accurate result it should be human handed anyhow rather then provide wrong result. $\endgroup$
    – Milkmaid
    Nov 20, 2023 at 8:45
  • $\begingroup$ What will happen to tickets that are "too general" in the production system? Will you also be filtering them out from being categorised using the same tool there? $\endgroup$ Nov 20, 2023 at 9:04
  • $\begingroup$ In prod to be only the final model. "Too general" to be decided by the main model either by defining special category or I will expect inaccurate classification eg. 30% cat1, 25% cat2 and the rest distributed between the rest categories which will does not go over the defined threshold. But indeed multiple models can be used in series as well. $\endgroup$
    – Milkmaid
    Nov 20, 2023 at 9:55

1 Answer 1

2
$\begingroup$

When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process?

No, this is backwards for classifiers such as the one you are training.*

When training any model, it is usually important to expose it to the same kind of data that it will be operating on later.

However, even more important than that, when training any model for a purpose, you first take some steps to turn the purpose into something you can assess. Typically a metric that measures the value of success or cost of errors that will occur in real-world scenario.

So your starting point should be some test data that is representative of production data, and a way to assess how well your overall approach is working with that data. Ideally, this is not the same as your cross-validation data (which you may use 100s of times to tune individual models), but a separate set to determine whether one of your models is good enough to release.

After that, you have some freedom to make complex pipelines or adjust the training data, and experiment with ideas for creating more sophisticated models. Remember with the final test, to test the entire pipeline end-to-end.

In comments you suggest that the small refiner model will only be used on training dataset and not in production:

In prod to be only the final model. "Too general" to be decided by the main model either by defining special category or I will expect inaccurate classification eg. 30% cat1, 25% cat2 and the rest distributed between the rest categories which will does not go over the defined threshold.

Your problem here is that your refiner will have removed all the "too general" items from the training data. So the main model will not have examples to learn how to assign to that category, and you will have no control during training whether or not there is the inaccurate classification that you are expecting (the main model could equally well classify inaccurately with high confidence on data that is far outside its experience).

However, if you follow the advice above and set a metric of hold-out production data, you can at least give your ideas a try with some confidence that you will understand whether or not they have worked.

You also suggest:

But indeed multiple models can be used in series as well.

I recommend this pipeline approach, over trying to second-guess what the classifier will do with types of data that are significantly different from those it has been trained on. The end result may be two specialised models that work in series. Again you should set up your test data and test metric first. There could be little to no improvement over the simpler single model, in which case you may want to reject the pipeline approach as too complicated.


* There may be some scenarios in which automated quality control refinement is useful. For instance training a generative model that you want to only output high quality text (for some definition of "high quality"). However, the situation presented in your question does not seem to fit this.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .