Can the (sparse) categorical cross-entropy be greater than one?

Question

I am using AlexNet CNN to classify my dataset which contains 10 classes and 1000 data for each class, with 60-30-10, splits for train, validation, and test. I used different batch sizes, learning rates, activation functions, and initializers. I'm using the sparse categorical cross-entropy loss function.

However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8. Is it normal? If not, how can I solve this?

Answer 1

Both the sparse categorical cross-entropy (SCE) and the categorical cross-entropy (CCE) can be greater than $1$. By the way, they are the same exact loss function: the only difference is really the implementation, where the SCE assumes that the labels (or classes) are given as integers, while the CCE assumes that the labels are given as one-hot vectors.

Here is the explanation with examples.

Let $(x, y) \in D$ be an input-output pair from the labelled dataset $D$, where $x$ is the input and $y$ is the ground-truth class/label for $x$, which is an integer between $0$ and $C-1$. Let's suppose that your neural network $f$ produces a probability vector $f(x) = \hat{y} \in [0, 1]^C$ (e.g. with a softmax), where $\hat{y}_i \in [0, 1]$ is the $i$th element of $\hat{y}$.

The formula for SCE is (which is consistent with the TensorFlow implementation of this loss function)

$$ \text{SCE}(y, \hat{y}) = - \ln (\hat{y}_{y}) \label{1}\tag{1}, $$

where $\hat{y}_{y}$ is the $y$th element of the output probability vector $\hat{y}$ that corresponds to the probability that $x$ belongs to class $y$, according to $f$.

Actually, the equation \ref{1} is also the definition of the CCE with one-hot vectors as targets (which behave as indicator functions). The only difference between CCE and SSE is really just the representation of the targets, which can slightly change the implementation under the hood. Moreover, note that this is the definition of the CE for only $1$ training pair. If you have multiple pairs, you have to compute the CE for all pairs, then average these CEs (for a reference, see equation 4.108, section 4.3.4 Multiclass logistic regression of Bishop's book PRML).

Let's have a look at a concrete example with concrete numbers. Let $C=5$, $y = 3$, $\hat{y} = [0.2, 0.2, 0.1, 0.4, 0.1]$, then the SCE is

\begin{align} \text{SCE}(y, \hat{y}) &= - \ln (0.4) \approx 0.92, \end{align}

If $\hat{y} = [0.2, 0.2, 0.2, 0.1, 0.3]$, so $\hat{y}_{y} = 0.1$, and we still have $y = 3$, then the CCE is $2.3 > 1$.

You can execute this Python code to check yourself.

import numpy as np
import tensorflow as tf # Install TensorFlow 2.3!

y = 1
y_true = [3] # sparse label (integer)
y_true2 = [0, 0, 0, 1, 0] # one-hot vector

for y_y in [0.4, 0.1]:
    sce_np = -(y * np.log(y_y))
    print("SCE (NumPy) =", sce_np)

y_preds = [[0.2, 0.2, 0.1, 0.4, 0.1],
           [0.2, 0.2, 0.2, 0.1, 0.3]]
for y_pred in y_preds:
    sce_tf = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
    cce_tf = tf.keras.losses.categorical_crossentropy(y_true2, y_pred)

    print("SCE (TensorFlow) =", sce_tf)
    print("CCE (TensorFlow) =", cce_tf)

To answer the following question more directly.

However, while training, my loss value is greater than one (almost equal to 1.2) in the first epoch, but until epoch 5 it comes near 0.8. Is it normal? If not, how can I solve this?

Yes. It can happen, as explained above. (However, this does not mean that you do not have mistakes in your code.)