Single label classification into hierarchical categories using a neural network

Question

I am working on a classification problem into progressive classes. In other words, there is some hierarchy of categories in such a way, that A < B < C, e.g. low, medium, high, very high. What loss function and activation function for the output layer should I use to take advantage of the class hierarchy, so that true A and predicted C is penalized more than true A and predicted B?

My ideas are:

1) To assign some value to each category, use one output unit with the sigmoid activation and RMS loss function. Then to assign each class to an interval, e.g. 0-033 - class A, 0.33-0.66 class B 0.66-1 - class C. It seem to do the trick, but can favor the extreme categories over the middle ones.

2) Use K softmax output units, integer labels instead of one-hot encoded and the sparse categorical crossentropy loss function. In this case I am not sure how exactly sparse categorical crossentropy works and if it really takes into account the hierarchy.

Answer 1

In my opinion, the problem you pose is best described as an ordinal classification problem, rather than a hierarchical classification problem. There are a number of approaches (besides ordinal loss functions) to address this problem discussed in the linked review article.

What loss functions should you use? There are a number of order-aware loss functions that have been described, such as mean absolute deviation (MAD), mean squared error (MSE), and ordinal crossentropy loss. Which one works best for your dataset must be empirically determined.

Sparse categorical crossentropy loss will not by itself take order into consideration. However, as I mentioned, a version of crossentropy loss has been described, which does take order into account (see article).

Don't forget to re-code your target variable such so that ordinal methods can be applied (1, 2, 3, ...).