I am trying to look through a code of the transformer model from Pytorch. However, I do not understand why batch size needs to multiply with cross-entropy loss given that loss is calculated based on data at a given timestep.
This is from the line: "total_loss += batch_size * criterion(output_flat, targets).item()"
This is the section of code:
def evaluate(model: nn.Module, eval_data: Tensor) -> float: model.eval() # turn on evaluation mode total_loss = 0. src_mask = generate_square_subsequent_mask(bptt).to(device) with torch.no_grad(): for i in range(0, eval_data.size(0) - 1, bptt): data, targets = get_batch(eval_data, i) batch_size = data.size(0) if batch_size != bptt: src_mask = src_mask[:batch_size, :batch_size] output = model(data, src_mask) output_flat = output.view(-1, ntokens) total_loss += batch_size * criterion(output_flat, targets).item() return total_loss / (len(eval_data) - 1)