# How should I penalize the model proportionally to the error?

I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?

I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?

Really you should not, because the symbols used (Arabic numerals) do not have direct relation to quantity in the same way e.g. tally counts or dots do. They are good candidates for classification, and despite the conventional mapping to quantity when you read them, the symbols themselves are poor candidates for regression, because for instance the symbols $$3$$ and $$4$$ do not differ in a way that captures quantity in any intuitive manner.

However, if you are keen to do this, it is relatively simple to construct a suitable loss function in most auto-differentiating frameworks. You will need to read up on how to do so. For instance, here is a Stack Overflow answer explaining where to start with writing custom loss function in Keras.

In order for your loss function to work, it will need to be differentiable and smoothly changing as predictions get better. That rules out using any form of argmax for the current prediction. If you want to stick with softmax for the final layer, then I suggest using a mean squared error against the expected prediction, e.g. if $$d_i$$ is the numerical digit for example $$i$$ and $$y_{i,j}$$ is the ground truth expressed as a one-hot vector, where $$i$$ is the example and $$j$$ is the digit class, then $$\hat{y}_{i,j}$$ is the probability predicted by your model. You could use $$\hat{d}_i = \sum_{j=0}^9 j\hat{y}_{i,j}$$ for the expected value and MSE loss of $$\mathcal{L}(d_i,\hat{d}_i) = \frac{1}{2}(\hat{d}_i - d_i)^2$$

You can also use a weighted sum of the MSE loss and cross entropy loss as your final loss, with the balance between the two losses being a new hyperparameter of your model.

Note this solution makes $$0$$ close to $$1$$ but far away from $$9$$. If you want the digits to be considered close on a cycle (e.g. $$8$$ is closer to $$1$$ than it is to $$4$$) the you will need something more creative.

Whilst I don't think this will help you discover any improvements to MNIST classification, combining two or more loss functions to achieve a more complex goal can be really useful sometimes, so it is a skill worth practicing.