How to train a sample weight model for another ML model?

Question

I'm trying to train a ML model, however the predictability of the different samples varies, i.e. some samples are inherently much harder to predict/estimate than others. Poorer predictions for these samples is fine and expected. However, after a few iterations these samples dominate the loss, resulting in no further improvement in the prediction of the other samples.

My plan is to use sample weights to address this, however as I do not have these sample weights and they are a function of another set of features, I was thinking of training another ML model to estimate the sample weights. My thinking is to train these two models together, i.e. using a loss function of the form $L = w * MAE(y, \hat{y})$. However, this will just result in the weight model predicting $w=0$ as it minimizes the loss function.

What are some approaches to prevent the weights from just going to zero during training? I'm guessing there's is quite a few, I just can't seem to find them (unsure what to search for).

The only one that has jumped to mind is to normalize the weights such that the sum to some constant, but I don't think this would work with batch-based gradient descent?

Thanks!

Answer 1

Sample Reweighting

There's actually been a good amount of work on sample reweighting, namely learning a set of weights for each datapoint in your training data. However, the goal of these algorithms seems more to be to correct for dataset bias rather than outliers. For example, if your training data is biased towards a certain class, then you'd want to underweight samples in that class.

Past work avoids the problems you mentioned by optimizing the weights with respect to a held-out validation set. Importantly, the weights learned are for the training set, that way, you can't just set all the weights to zero because then your model wouldn't learn anything and your validation set performance would drop.

Meta-learning

"Learning to Reweight Examples for Robust Deep Learning" is one well-cited paper that does this.

The intuition behind their algorithm is:

the best example weighting should minimize the loss of a set of unbiased clean validation examples that are consistent with the evaluation procedure

They do this through approximate meta-learning. The weights for each sample in the training set are learnable parameters. Vanilla meta-learning would involve two optimization loops: for each update you make to the weights, you'd need to do a number of iterations for the actual training set. They approximate this by just looking at gradients of the sample weights wrt the validation loss in a single iteration. A large negative gradient would mean that increasing the weight of that sample would lead to a large decrease in the validation loss, so you should weigh that sample more heavily. Note that they do normalize the sample weights here such that weights over each batch sum to one.

There's a ton of literature on sample weighting, including techniques that don't use meta-learning (e.g., Just Train Twice, which weighs points that models misclassify more heavily), or involve learning a model to predict weights. I'd recommend you follow the citation trail, or look sample reweighting for more work on this topic.

Outlier detection and removal

You may also want to consider using a different technique, rather than sample reweighting, especially because the goal of the technique tends to be improving robustness/generalization and reducing the impact of training set bias. In fact, these techniques often weigh difficult samples more highly.

This paper looks at this issue with Just Train Twice, where unlearnable or OOD samples end up getting weighed heavily. They tackle this by detecting and removing outliers based on the Mahalanobis distance on the hidden representations of the network (based on this paper). Other work involves looking at the softmax probabilities of the predicted class or training a classifier to predict OOD samples.

There's also a ton of work on this, and the papers I mentioned are by no means comprehensive. I'd recommend taking a look at work on OOD sample detection, outlier detection, or handling mislabeled data.