I would like to ask a question about the relationship of accuracy with the loss function.
My experiment is a multiclass text classification problem, and I have built a Keras neural network to tackle it. My labels are something like
array([array([0, 0, 0, 0, 0, 1, 0, 1]), array([0, 1, 1, 0, 0, 0, 0, 1])])
For the final output layer I use the 'sigmoid' activation function and for loss the 'binary crossentropy', however, I am a bit confused about the metric. I am using the F1_score metric because Accuracy it's not a metric to count on when there are many more negative labels than positive labels. So, since the problem is multilabel classification shall I use the multi-label mode like tfa.metrics.F1_score(mode="micro"). Is that correct? or since I use binary_crossentropy and sigmoid activation function should I use the standard binary f1-score because every label-tag Is independent to the others and has a different Bernoulli distribution?
I would really like to get your input on this. My humble opinion is that I should you the binary standard mode of binary f1-score and not the multi-label micro approach even though my experiment is multi-label text classification.
My current approach (using micro F1-score since my y_train is Multi-label
model_for_pruning.compile(optimizer='adam', loss='binary_crossentropy', metrics=[tfa.metrics.F1Score(y_train.shape[-1], average="micro")])
My alternative approach (based on the binary_crossentropy and the sigmoid activation function, despite I have multi-label y_train)
model_for_pruning.compile(optimizer='adam', loss='binary_crossentropy', metrics=[tfa.metrics.F1Score(y_train.shape[-1], average=None)])
Τhe reason why I use sigmoid and not softmax as the output layer
relevant link Why Sigmoid and not Softmax in the final dense layer? In the final layer of the above architecture, sigmoid function as been used instead of softmax. The advantage of using sigmoid over Softmax lies in the fact that one synopsis may have many possible genres. Using the Softmax function would imply that the probability of occurrence of one genre depends on the occurrence of other genres. But for this application, we need a function that would give scores for the occurrence of genres, which would be independent of occurrences of any other movie genre.
relevant link 2 Binary cross-entropy rather than categorical cross-entropy. This may seem counterintuitive for multi-label classification; however, the goal is to treat each output label as an independent Bernoulli distribution and we want to penalize each output node independently.
Please check my reasoning behind this and I would be happy if we can contradict this explanation. To better explain my experiment, I want to predict movie genres. So a movie can belong to 1 or more genres ['Action', 'Comedy', 'Children'], so when I use softmax the probability sum to 1, while when I use sigmoid its single probability of a class has a range between (0,1). Thus, if the predictions are correct the genres with the highest probabilities are those assigned to the movie. So imagine that my vector of prediction probabilities are something like [0.15, 0.12, 0.54, 0.78, 0.99] sum()> 1, and not something like [0.12, 0.43, 11, 0.32, 0.01, 0.01) sum() = 1.