What do you mean by "updating based on a training example/batch" in Gradient Descent?

Question

My understanding is this: When doing Stochastic Gradient Descent over a neural network, in every epoch, we run $n$ iterations (where the dataset has $n$ training examples) and in every iteration, we take a random sample and update the parameters wrt the sample.

However, in batch gradient descent, we take the whole dataset every epoch, and update the parameters wrt the batch. I have the following questions:

1. Why do we need to compute the loss function every time, especially if the value of the loss has no importance as such in the backprop process? Is it just to ensure that the loss decreases over time? How can you propagate the error back when the value of L is of no significance?
2. What exactly does updating wrt a "batch" mean? What would you take the input vector (required to compute gradients for the first set of weights) as? I assume that the loss is taken to be the average of the losses for the entire batch

Answer 1

In batch gradient descent computing the loss function every time serves several purposes. While the value of the loss itself may not directly affect the backpropagation process, as you said monitoring the loss over time is essential to ensure that the optimization process is converging. Also many optimization algorithms such as learning rate schedules or adaptive learning rate methods adjust the learning rate or other hyperparameters based on the behavior of the loss function during training. Therefore, computing the loss function is crucial for determining how the learning rate should be updated. Finally, monitoring the loss function can also be used for early stopping, a technique where training is halted if the loss on a validation set stops decreasing or starts increasing, which helps prevent the usual overfitting of neural networks and wasting computational resources.

Updating parameters with respect to a batch means computing the gradients of the (average) loss function with respect to the parameters using all examples in the batch and then updating the parameters accordingly. This process is repeated for each batch during each epoch of training and in your case each batch could simply mean the entire training dataset.

The input vector used to compute gradients for the first set of weights is a matrix whose row represents an individual example from the batch and whose columns represent the features of that example, and the number of features would typically determine the number of input nodes of your network. During training the neural network processes this entire batch at once with operations including forward and backward passes performed in a vectorized or parallelized manner for efficiency.