why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?

Question

I am trying to look through a code of the transformer model from Pytorch. However, I do not understand why batch size needs to multiply with cross-entropy loss given that loss is calculated based on data at a given timestep.

This is from the line: "total_loss += batch_size * criterion(output_flat, targets).item()"

This is the section of code:

def evaluate(model: nn.Module, eval_data: Tensor) -> float:
    model.eval()  # turn on evaluation mode
    total_loss = 0.
    src_mask = generate_square_subsequent_mask(bptt).to(device)
    with torch.no_grad():
        for i in range(0, eval_data.size(0) - 1, bptt):
            data, targets = get_batch(eval_data, i)
            batch_size = data.size(0)
            if batch_size != bptt:
                src_mask = src_mask[:batch_size, :batch_size]
            output = model(data, src_mask)
            output_flat = output.view(-1, ntokens)
            total_loss += batch_size * criterion(output_flat, targets).item()
    return total_loss / (len(eval_data) - 1)

Answer 1

That is because the default reduction for the CEP loss calculation is mean. Hence to find the true average across all batches, you first multiply by the batch size and then divide by total number of data points. Note this only matters when all the batches are not equal, i.e. total number of data points do not perfectly divide batch size. Else you can simply add all the batch losses and divide by number of batches.