What is the difference between batch and mini-batch gradient decent?

Question

I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent.

Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?

Answer 1

It is really simple.

In gradient descent not using mini-batches, you feed your entire training set of data into the network and accumulate a cost function based on this full set of data. Then you use gradient descent to adjust the network weights to minimize the cost. Then you repeat this process until you get a satisfactory level of accuracy. For example, if you have a training set consisting of 50,000 samples, you would feed all 50,000 samples along with the 50,000 labels into the network, then perform gradient descent and update the weights. This is a slow process because you have to process 50,000 inputs to do just one step of gradient descent.

To make things go faster instead of running all 50,000 inputs through the network, you split up the training set into "batches". For example, you could break the training set up into 50 batches each containing 1000 samples. You would feed the network the first batch of 1000 samples, accumulate the loss value then perform gradient descent and adjust the weights. Then you feed in the next batch of 1000 samples and repeat the process. So, now, instead of only getting one step of gradient descent for 50,000 samples, you get 50 steps of gradient descent. This method of using batches leads to a much faster convergence of the network.

Answer 2

The other answer provides a correct description of what is often called "gradient descent" and "mini-batch gradient descent", respectively, but doesn't clarify the terminology used, so let me add a few notes about that.

From my experience, "batch GD" and "mini-batch GD" can refer to the same algorithm or not, i.e. some people may use "batch GD" and "mini-batch GD" interchangeably, but other people may use "batch GD" to refer to what the author of the other answer calls "gradient descent not using mini-batches", i.e. you use all training data before performing a GD step, which is sometimes just called "gradient descent" (as I wrote above).

Answer 3

Batch Gradient Descent(BGD):

• can do gradient descent with a whole dataset, taking only one step in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens only once in one epoch which means model's parameters are updated only once in one epoch.
• 's pros:
  • The computation is stable(not fluctuated).
  • It's strong in noise(noisy data). *Noise(noisy data) means outliers and anomalies.
  • It gets an accurate value.
• 's cons:
  • It's not good at a large dataset because it takes much memory slowing down the computation.
  • It needs the repreparation of a whole dataset if you want to update a model.
  • It cannot easily escape local minima or saddle points.

Mini-Batch Gradient Descent(MBGD):

• can do gradient descent with splitted dataset(the small batches of a whole dataset) one small batch by one small batch, taking the same number of steps as the small batches of a whole dataset in one epoch. For example, the whole dataset which has 100 samples(1x100) is splitted into 5 small batches(5x20), then gradient descent happens 5 times in one epoch which means model's parameters are updated 5 times in one epoch.
• 's pros:
  • It's good at a large dataset because it takes small memory not slowing down the computation.
  • It's good at online learning. *Online learning is the way which a model incrementally learns from a stream of dataset in real-time.
  • It doesn't need the repreparation of a whole dataset if you want to update a model.
  • It can more easily escape local minima or saddle points than BGD.
• 's cons:
  • The computation is less stable than BGD.
  • It's less strong in noise(noisy data) than BGD.
  • It can more often escape local minima or saddle points than BGD.