New answers tagged

2

They have a few similarities, but they are quite different. Let me first give you a general description of both approaches/algorithms, so that you start to get a sense of their differences and similarities. Description Gradient descent (GD) can be applied to solving any optimization problem where your loss (aka cost or objective) function is differentiable ...


0

How many optimisation iterations are performed on a mini-batch? Just one, as you suspected. then how does an optimisation algorithm like adam work which uses past gradient information? It uses the gradient estimates from each mini-batch as its input sequence. It seems strange since then gradients from past mini-batches are being used to minimise the loss ...


1

In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have special dimension (usually corresponding to the 0 axis) which numerates the example in the batch. Look for example in the PyTorch documentation of Conv2d or Tensorflow documentation of Conv2d. The same is true for any Layer - Linear, MultiheadAttention, RNN. All samples from ...


0

What happens in mini-batches is not very different from the way updates are made in batch gradient descent, only the number of samples is different. In mini-batch, you process all the data in the batch, and the update happens after that. It is detailed in this video after 6:11.


Top 50 recent answers are included