0
$\begingroup$

My Python source code uses TensorFlow and Keras to implement a neural network.

The Keras source code uses something called "categorical cross-entropy" and "categorical accuracy metric". I have searched a lot of books on NN theory, and no one talks about these two specific terms. Yes, they talk about "cross-entropy" and "accuracy metric" but there are no mentions of "categorical ...".

N.B. These terms can be found only in the so-called "Hands-on" books.

Can anyone please supply me with authentic references on these two specific terminologies?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
2
+50
$\begingroup$

Categorical just means that we will conduct multiclass classification. The output of the classifier is a binary vector. Each entry $x_i$ in the binary vector is a prediction whether or not the input is part of class $C_i$.

In that sense, categorical accuracy introduces nothing new: it is just the accuracy of a multiclass classifier. On the other hand, categorical cross-entropy refers to the joint entropy: $H(X_1, X_2) = - \sum{p(x_1,x_2)\log_2p(x_1, x_2)}$, where each random variable $X_i$ expresses whether or not the input is to be classified into class $C_i$. Kevin Murphy's book "Probabilistic Machine Learning: An introduction" is a great source of reference for many topics of machine learning, including cross-entropy and joint entropy.

The random variables could be mutually exclusive or not, depending on the problem.

  • If they are mutually exclusive, we would allow only one of the classes to have value 1 during training and use the softmax activation function.
  • If they are not mutually exclusive, we would allow multiple classes to have value 1 during training and use the sigmoid activation function.
Improve this answer
$\endgroup$
2
  • $\begingroup$ what about the categorical accuracy metric? $\endgroup$
    – user366312
    Feb 16, 2022 at 16:01
  • $\begingroup$ I edited the answer to mention categorical accuracy too. Basically, categorical accuracy is not a separate concept, so you would have a hard time finding it in literature. $\endgroup$
    – devidduma
    Feb 16, 2022 at 21:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .