I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent.
Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?
Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.Sign up to join this community
It is really simple.
In gradient descent not using mini-batches, you feed your entire training set of data into the network and accumulate a cost function based on this full set of data. Then you use gradient descent to adjust the network weights to minimize the cost. Then you repeat this process until you get a satisfactory level of accuracy. For example, if you have a training set consisting of 50,000 samples, you would feed all 50,000 samples along with the 50,000 labels into the network, then perform gradient descent and update the weights. This is a slow process because you have to process 50,000 inputs to do just one step of gradient descent.
To make things go faster instead of running all 50,000 inputs through the network, you split up the training set into "batches". For example, you could break the training set up into 50 batches each containing 1000 samples. You would feed the network the first batch of 1000 samples, accumulate the loss value then perform gradient descent and adjust the weights. Then you feed in the next batch of 1000 samples and repeat the process. So, now, instead of only getting one step of gradient descent for 50,000 samples, you get 50 steps of gradient descent. This method of using batches leads to a much faster convergence of the network.
The other answer provides a correct description of what is often called "gradient descent" and "mini-batch gradient descent", respectively, but doesn't clarify the terminology used, so let me add a few notes about that.
From my experience, "batch GD" and "mini-batch GD" can refer to the same algorithm or not, i.e. some people may use "batch GD" and "mini-batch GD" interchangeably, but other people may use "batch GD" to refer to what the author of the other answer calls "gradient descent not using mini-batches", i.e. you use all training data before performing a GD step, which is sometimes just called "gradient descent" (as I wrote above).