Binary mode or Multi-label mode is correct when using binary crossentropy and sigmoid output function on multi-label classification

Question

I would like to ask a question about the relationship of accuracy with the loss function.

My experiment is a multiclass text classification problem, and I have built a Keras neural network to tackle it. My labels are something like

array([array([0, 0, 0, 0, 0, 1, 0, 1]), array([0, 1, 1, 0, 0, 0, 0, 1])])

For the final output layer I use the 'sigmoid' activation function and for loss the 'binary crossentropy', however, I am a bit confused about the metric. I am using the F1_score metric because Accuracy it's not a metric to count on when there are many more negative labels than positive labels. So, since the problem is multilabel classification shall I use the multi-label mode like tfa.metrics.F1_score(mode="micro"). Is that correct? or since I use binary_crossentropy and sigmoid activation function should I use the standard binary f1-score because every label-tag Is independent to the others and has a different Bernoulli distribution?

I would really like to get your input on this. My humble opinion is that I should you the binary standard mode of binary f1-score and not the multi-label micro approach even though my experiment is multi-label text classification.

My current approach (using micro F1-score since my y_train is Multi-label

model_for_pruning.compile(optimizer='adam',
                          loss='binary_crossentropy',
                          metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average="micro")])

My alternative approach (based on the binary_crossentropy and the sigmoid activation function, despite I have multi-label y_train)

model_for_pruning.compile(optimizer='adam',
                          loss='binary_crossentropy',
                          metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average=None)])

Τhe reason why I use sigmoid and not softmax as the output layer

relevant link Why Sigmoid and not Softmax in the final dense layer? In the final layer of the above architecture, sigmoid function as been used instead of softmax. The advantage of using sigmoid over Softmax lies in the fact that one synopsis may have many possible genres. Using the Softmax function would imply that the probability of occurrence of one genre depends on the occurrence of other genres. But for this application, we need a function that would give scores for the occurrence of genres, which would be independent of occurrences of any other movie genre.

relevant link 2 Binary cross-entropy rather than categorical cross-entropy. This may seem counterintuitive for multi-label classification; however, the goal is to treat each output label as an independent Bernoulli distribution and we want to penalize each output node independently.

Please check my reasoning behind this and I would be happy if we can contradict this explanation. To better explain my experiment, I want to predict movie genres. So a movie can belong to 1 or more genres ['Action', 'Comedy', 'Children'], so when I use softmax the probability sum to 1, while when I use sigmoid its single probability of a class has a range between (0,1). Thus, if the predictions are correct the genres with the highest probabilities are those assigned to the movie. So imagine that my vector of prediction probabilities are something like [0.15, 0.12, 0.54, 0.78, 0.99] sum()> 1, and not something like [0.12, 0.43, 11, 0.32, 0.01, 0.01) sum() = 1.

Answer 1

Since you have a multiclass classification problem rather than a binary classification problem (i.e. a two-class problem), I recommend to adjust your architecture and use softmax instead of sigmoid as final activation function and categorcal_crossentropy instead of binary_crossentropy.

Softmax will ensure all your outputs are valid probabilities. This is crucial in calculating the loss of your model. Using sigmoid will only ensure this if you have binary classes. Further categorical crossentropy is used for multiclass problems, again this is to make sure you calculate the correct loss. This is just a general model characteristic:
binary -> sigmoid & binary_crossentropy
multi-class -> softmax & (sparse_)categorical_crossentropy

To answer you original question, I think you should use average = micro, macro or weighted when calculating the F1-score. This depends on what you like your model to put more emphasis on and how severe your class imbalances are.