Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

Question

While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent.

For this question, assume that your dataset has $n$ training samples and we divided it into $k$ batches with $\dfrac{n}{k}$ samples in each batch. So, it can be easily understood the word "batch" is generally used to refer to a portion of the dataset rather than the whole dataset.

In batch gradient descent, we pass all the $n$ available training samples to the network and then calculates the gradients (only once). We can repeat this process several times.

In mini-batch gradient descent, we pass $\dfrac{n}{k}$ training samples to the network and calculates the gradient. That is, we calculate the gradient once for a batch. We repeat this process with all $k$ batches of samples to complete an epoch. And we can repeat this process several times.

In stochastic gradient descent, we pass one training sample to the network and calculates the gradient. That is, we calculate the gradient once for iteration. We repeat this process with all $n$ times to complete an epoch. And we can repeat this process several times.

Batch gradient descent can be viewed as a mini-batch gradient descent with $k = 1$ and stochastic gradient descent can be viewed as a mini-batch gradient descent with $k = n$.

Am I correct regarding the usage of terms in the context provided above? If wrong then where did I go wrong?

If correct, I am confused about the usage of the word "batch" in "batch gradient descent". In fact, we do not need the concept of batch in batch gradient descent since we pass all the training samples before calculating gradient. In fact, there is no need for batch gradient descent to partition the training dataset into batches. Then why do we use the word "batch" in batch gradient descent? Similarly, we are using the word "mini-batch" in "mini-batch gradient descent". In fact, we are passing a batch of samples before calculating the gradient. Then why it is called "mini-batch" gradient descent instead of "batch" gradient descent?

Answer 1

You are correct, but requires final words:

In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i.e. smooth error).

On the other hand, in the Stochastic GD, we take one training sample to go one step towards the optimum, then repeat the latter for every training sample, hence updating the parameters once per sample sequentially in every epoch (no average here). As you can expect, the training will be noisy and the error will be fluctuating.

Lastly, the mini-batch GD, is somehow in between the first two methods, that is: the average of a different portion of the data, every time. This method would take the benefits of the previous two, not so noisy, yet can deal with less smooth error manifold.

---

Personally, I memorize them in my mind by creating the following map:

1. Batch GD ≡ Average of All per Step ≡ More suitable for Convex Problems at the Risk of Converging directly to Minima = Heavyweight.
2. Stochastic GD ≡ Fluctuating & Noisy ≡ Converges on the Long Run especially for Large Dataset ≡ Lightweight but Slower ≡ No Vectorization Possible (because one sample per time).
3. Mini-Batch GD ≡ Portion of Data per Step ≡ Mixture of Stochastic and Batch GD ≡ Less Fluctuation + Would work for Less Smooth Error Manifolds ≡ Faster Computation.

---

Regarding the naming convention, I would understand the word "Batch" as a "Set" or "Collection", hence the whole "Dataset". Consequently, "Mini" would go with the flow, to mean a "Part of the Set".