Regression loss conditioned by the ground-truth values

Question

I'm working on a regression problem with a CNN in which the input is a single image, and the output is an angle in degrees (which determines a specific measure related to the image).

Sometimes, the model fails to retrieve the output accurately (for output angles wider than 20°). By analyzing the data, I can suspect there is a problem of imbalance since there are a lot of training samples with outputs between -20° and 20° but very few for wider angles (the wider the angle, the fewer the examples).

There is no possibility to generate more data to balance the training set artificially. I want to try a robust loss function focusing more on wide angles.

My model is typically trained with MSE, but I would like to implement a loss function in pytorch, which is linear for ground-truth values lower than 20° (in absolute value) and becomes quadratic for greater values. This is similar to the Huber Loss, but 1) the conditions would be the opposite. 2) The condition of the Huber function would be based on the GT instead of the absolute difference between the GT and the output.

So, my question is whether this solution would make sense from an ML point of view since I haven't seen something like this in ML literature. Or is there maybe a more "standard" way to face this problem ? (apart of data augmentation)

Thanks in advance.

Answer 1

Your suggestion should work to focus the ML more on larger angle examples.

You may want to try a slightly simpler approach of weighting the loss (or the resulting gradient) by a factor depending on rarity of the angle in the dataset. That would give you greater control, and a mathematical rationale for choosing weight values (usually inverse of frequency in the dataset for each group). This is a common weighting method for imbalanced classes, that you would be adapting for your regression problem - your could start by bucketing your data into 5 degree "classes" with an outermost class of e.g. everything greater than 40 degrees, and then assign a weighting to each bucket.

In terms of end results, it doesn't usually matter what loss function you end up using to train your model. What matters is how well the resulting model works in production environments where its predictions are being used for something practical.

Ultimately you should base decision of what the "best" model is using some metric on untrained data, and ideally you set that metric before you go about trying different approaches. Otherwise you could get lost looking at errors you don't like, but have no way to assess if fixing them and dealing with the inevitable other errors this will produce ends up with a better model. The metric should, as much as possible, reflect the actual costs to your project for inaccuracy.

There is no free lunch, and whatever approach you take you will need to concern yourself with:

• Focusing on fitting better to rarer data means potentially fitting worse to the more common data.

• Focusing on fitting better to rarer data means larger risk of overfitting in general.

• Techniques for weighting or adaptive loss functions add more hyperparameters to tune.