Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge and training will be very slow. With the advent of deep learning, training size is in millions and computation using all training examples is very impractical and very slow.
This is where, two optimization techniques became prominent.
- Mini-Batch Gradient Descent
- Stochastic Gradient Descent (SGD)
In mini-batch gradient descent, you use a batch size that is considerably less than total no. of training examples and update your weights after passing through these examples.
In stochastic gradient descent, you update the weights after passing through each training example.
Coming to advantages and disadvantages of the three methods we discussed.
Batch gradient descent gradually converges to the global minimum but
it is slow and requires huge computing power.
Stochastic gradient descent converges fast but not to the global
minimum, it converges somewhere near to the global minimum and
hovers around that point, but doesn't converge ever to the global
minimum. But, the converged point in Stochastic gradient descent
is good enough for all practical purposes.
Mini-Batch gradient is a trade-off the above two methods. But, if you
have a vectorized implementation of the weights updation and you
are training with a multi-core setup or submitting the training to
multiple machines, this is the best method both in terms of time for
training and convergence to global minimum.
You can plot the cost function, w.r.t the no. of iterations to understand the difference between convergence in all the 3 types of gradient descent.
Batch gradient descent plot falls smoothly and slowly and gets stabilized and
gets to global minimum.
Stochastic gradient descent plot will have oscillations, will fall
fast but hovers around global minimum.
These are some blogs where there is detailed explanation of advantages, disadvantages of each method and also graphs of how cost function changes for all the three methods with iterations.