Let's suppose that we have a multi-class classification problem with 5 classes: 0, 1, 2, 3, 4. The order is not random, they are neighbors. For example, imagine that a labelling is 1. If the prediction for it is 1, it is the best. If the prediction is 0 or 2, it is also good, but not the best. If the prediction is 3 is worse, while if the prediction is 4, it is the worst.

I want neither the classical cross entropy which simply penalizes any misclassification equally, nor the weighted one which simply penalizes more the classes with less samples in an imbalanced dataset. I want to specifically and per-pair penalize any of the possible (25 in my case) combinations of prediction-label.

So I have 4 specific questions:

  1. How is this so much customized loss called? What do I have to google and what do you suggest me?

  2. What about the custom metric? I am talking about a custom metric for monitoring during the training, not in meta-training. I suppose it is something more advanced than the F1 score or the Fβ score?

  3. Can these weights be somehow trainable? For example, let's say, by using Gradient Reversal layers?

  4. Is this problem really multi-class classification or should I switch to something else? Generally speaking, is it good that I focus on improving the loss and the custom metric for monitoring, because I feel my model (Transformer) is doing very well but needs full human help and customization, or I should totally focus on its architecture and the classical - default accuracy - cross-entropy are more than enough?



You must log in to answer this question.