How to define a loss function for multi-label problem?

Question

I have voice recordings which are labelled by not only a single label but multiple labels. Each voice recording corresponds to one of class labels within a set. In other words, the training instance is given a set of (or distribution over) candidate class labels and only one of the candidate labels is the correct one.

I wish to train a model that classifies which class label corresponds to each voice recording. Each one of my voice recordings is accompanied by a set of 10 potential labels (labels are always different), but it is unknown which label it is exactly (aside from a small sample where there is only one correct label).

This is due to the nature of where my data comes from: someone records a short voice message and then types the same message into a chat, however there will be slight delay between the two and in the meanwhile other chat messages arrive. Only one of the next 10 chat messages after the voice message is the correct one that corresponds to that voice message.

How would I define a loss function in this case?

Answer 1

Given your response in the comments, you are faced with a semi-supervised learning problem where you have a small set of data with ground truth labels, and a large set of data without ground truth labels.

If you look at a similar problem, predicting ImageNet label with only 10% of the ground truth labels, you can see that the best methods reach ~80% accuracy.

You can try to reproduce the approach of Big Self-Supervised Models Are Strong Semi-Supervised Learners, which achieves ~74% accuracy on ImageNet with only 1% of the labels.

Namely:

1. Unsupervised pre-training step. Train a deep and wide model on the unlabelled data. The idea is to minimize a contrastive loss using slightly modified versions of the training samples. The method they introduce is called SimCLRv2, which you can reuse in your problem by using data augmentation methods suited to audio samples. You can read on this topic in this Pytorch tutorial.
2. Fine tuning step. Fine tune the model obtained in step 1 on the small dataset for which you have the ground truth labels. To do this, replace the last few layers of your big unsupervised model with an MLP and train it with a classic cross entropy loss (just the MLP part, the rest of the model should be frozen).
3. [Optional] Distillation step. Use the model of step 2 to label the data for which you didn't have ground truth labels and train a smaller network in a supervised manner on these new labels (and only those, exclude the data for which you had the ground truth labels!).

I've simplified a lot the method described in the paper, to give you an idea of the general approach. Read the paper for details!

Answer 2

If I understand correctly, the training data are voice messages which contain one or more suggested labels for the class that the voice message belongs to. Even though there are multiple suggestions, each voice message only truly belongs to one class.

If so, you could define the output layer of the network to have 10 nodes (the number of classes), with each node outputting values between 0 and 1 using a sigmoid function.

The loss to base optimisation could be the MSE of the sum of the element-wise difference between the input vector say [1,1,1,0,0,0,0,0,0,0] (the first 3 classes are suggested) and the output vector of the network [0.78, 0.76, 0.65, 0.23 ... 0.11, 0.36].

Once the model is sufficiently trained, for prediction you could apply a softmax to the output vector and select the largest value as the class to select.