Are the training loss and validation loss plotted per sample or per batch?

Question

I am using a CNN to train on some data, where training size = 21700 samples, and test size is 653 samples, and say I am using a batch_size of 500 (I am accounting for samples out of batch size as well).

I have been looking this up for a long time now, but can't get a clear answer, but when plotting the loss functions to check for whether the model is overfitting or not, do I plot as follows

for j in range(num_epochs):
  <some training code---Take gradient descent step do wonders>
  batch_loss=0
  for i in range(num_batches_train):
       batch_loss = something....criterion(target,output)...
       total_loss += batch_loss
  Losses_Train_Per_Epoch.append(total_loss/num_samples_train)#and this is 

where I need help

Losses_Train_Per_Epoch.append(total_loss/num_batches_train)
and doing the same for Losses_Validation_Per_Epoch.
plt.plot(Losses_Train_Per_Epoch, Losses_Validation_Per_epoch)

So, basically, what I am asking is, should I divide by num_samples or num_batches or batch_size? Which one is it?

Answer 1

You want to compute the mean loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches!

In your case:

You have a training set of $21700$ samples and a batch size of $500$. This means that you take $21700/500 \approx 43$ training iterations. This means that for each epoch the model is updated $43$ times! So the way you compute your training loss, that is what you need to divide by.

Note: I'm not sure what exactly you're trying to plot but I'm assuming you want to plot the training losses and the validation losses

training_loss = []
validation_loss = []
training_steps = num_samples // batch_size
validation_steps = num_validation_samples // batch_size

for epoch in range(num_epochs):

    # Training steps
    total_loss = 0
    for b in range(training_steps):
        batch_loss = ...  # compute batch loss
        total_loss += batch_loss
    training_loss.append(total_loss / training_steps)

    # Validation steps
    total_loss = 0
    for b in range(validation_steps):
        batch_loss = ...  # compute batch validation loss
        total_loss += batch_loss
    training_loss.append(total_loss / validation_steps)

# Plot training and validation curves
plt.plot(range(num_epochs), training_loss)
plt.plot(range(num_epochs), validation_loss)

Another way would be to store the losses in a list and compute the mean. You can use this if you're not sure with what to divide.

...

for epoch in range(num_epochs):

    list_of_batch_losses = []  # initialize list that is going to store batch losses

    # Training steps
    for b in range(training_steps):
        batch_loss = ...  # compute batch loss
        list_of_batch_losses.append(batch_loss)  # store loss in a list

    epoch_loss = np.mean(list_of_batch_losses)
    training_loss.append(epoch_loss)

    ...

plt.plot(range(num_epochs), training_loss)