New answers tagged


They have a few similarities, but they are quite different. Let me first give you a general description of both approaches/algorithms, so that you start to get a sense of their differences and similarities. Description Gradient descent (GD) can be applied to solving any optimization problem where your loss (aka cost or objective) function is differentiable ...


How many optimisation iterations are performed on a mini-batch? Just one, as you suspected. then how does an optimisation algorithm like adam work which uses past gradient information? It uses the gradient estimates from each mini-batch as its input sequence. It seems strange since then gradients from past mini-batches are being used to minimise the loss ...


In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have special dimension (usually corresponding to the 0 axis) which numerates the example in the batch. Look for example in the PyTorch documentation of Conv2d or Tensorflow documentation of Conv2d. The same is true for any Layer - Linear, MultiheadAttention, RNN. All samples from ...


What happens in mini-batches is not very different from the way updates are made in batch gradient descent, only the number of samples is different. In mini-batch, you process all the data in the batch, and the update happens after that. It is detailed in this video after 6:11.

Top 50 recent answers are included