4
$\begingroup$

I was just doing a simple NN example with the fashion MNIST dataset, where I was getting 97% accuracy, when I noticed that I was using Binary cross-entropy instead of categorical cross-entropy by accident. When I switched to categorical cross-entropy, the accuracy dropped to 90%. I then got curious and tried to use binary cross-entropy instead of categorical cross-entropy in my other projects and in all of them the accuracy increased.

Now, I know that binary cross-entropy can be used in a multi-class, multi-label classification problem, but why is working better than categorical cross-entropy in a multiclass single label problem?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro Is relevant.

based on my reading when you have a NN and do Binary crossentropy on what you might call 'Linked category data' the accuracy can tend to be better than in a Categorical crossentropy model. The binary aspect implies the categories can undergo multiple splits before deciding on the exact category when the data is categorically splittable like this in a tree like hierarchy the accuracy can tend to be better.

Think of how difficult it would be to memorize the name of every type of clothing in someones wardrobe if each piece had its own special name. Vs if they had structurally relevant names like Upper/lower for category number one is it warn on your top half or bottom half. followed by inner or outer. It an inner or outer layer of clothing. learning such binary name/feature categories enables a more accurate model. If it was data unrelated in this way it would most likely not be as accurate. The binary model can take advantage learning such features while a muti categorical model I think assumes independence and tries to learn best the features of each group and gives out a prediction of how sure it falls in each category.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But, in that post, they say that binary cross entropy is only for 2 classes, meanwhile I have more in my classification problems $\endgroup$
    – joão correia
    Nov 9, 2019 at 23:33
  • $\begingroup$ I think you should imagine the output layer of a binary output is it true or not for each class vs the training done on a 3> softmax where it chose the highest value and back propagates based on being wrong or right. if you use binary crossentropy on a multi class output layer the loss is applied in a binary fashion, benefiting tree like classification data. Maybe reading these articles will help you. medium.com/data-science-bootcamp/… gombru.github.io/2018/05/23/cross_entropy_loss $\endgroup$
    – Michael Hearn
    Nov 9, 2019 at 23:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .