I am looking to try different loss functions for a hierarchical multi-label classification problem. So far, I have been training different models or submodels like multilayer perceptron ( MLP )branch inside a bigger model which deals with different levels of classification, yielding a binary vector. I have been also using Binary Cross Entopy(BCE) and summing all the losses existing in the model before backpropagating.

I am considering trying other losses like MultiLabelSoftMarginLoss and MultiLabelMarginLoss. What other loss functions are worth to try? Hamming loss perhaps or a variation? Is it better to sum all the losses and backpropagate or do multiple backpropagations?


A loss function, except in the most trivial case of one dimensional network output, is itself an aggregation. Aggregating loss functions is just an approach for producing a scalar loss from differences in multiple dimensions, and possibly an over-complicating one.

If the concept class is a hierarchy with three levels, then the loss function is an aggregation of differences occurring in one, two, or three dimensional space, based on whether the divisions in each category are identical throughout the hierarchy. If there are three label dimensions, they are statistically independent, and each has three states, then there are 27 categories that can be represented in three ternary outputs.

Trying several loss function options available in torch.nn.modules.loss is one way to find the best solution within PyTorch, but its possible to make the wrong choose because the hyper-parameters are not correctly set in the better choice. Or the labels could be improperly mapped to the poorly performing loss function.

It may be over-complicating the solution to embed one model in another when it is common to categorize in multiple dimensions with a single network. Take a simple case.

Let's say there is one category dimension of four age brackets, another of attire type (casual, business casual, formal, undetermined), and a third of eight cultural backgrounds including undetermined. The goal is to categorize head-to-toe images based on facial, hair, and clothing style queues with some specific reliability.

In this case, in addition to the appropriate convolution, pooling, and fully connected layers, the network output layer could have two binary outputs for age, two for attire, and three for cultural background.

The complexity occurs when the kinds of attire differ between cultural backgrounds. In this case, the five bits may determine a composite of attire types and cultural backgrounds, but developing separate networks, one for each culture type is probably overkill.

Mapping the dimensions of labels to the output vector should be the focus of attention. Entropy is a concept that applies to loss function selection, but the idea is to align negative entropy evenly with output vector components, so that information is equally distributed across the vector components.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.