11

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the dataset before working through it each time. or should I first calculate the desired changes for the entire lot of 1,000 samples, and only then start ...


7

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) only one training input (also known as data point, example, sample or observation) or multiple ones (a batch). However, the loss function to train the neural ...


6

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't completely answer the original question. In fact, I leave some unanswered questions at the end (that I will eventually answer). The gradient is a linear ...


5

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing. Note that, in the pseudocode, $j$ is not specified in detail. They do not, for example, have $j$ ranging from $0$ to the size of the minibatch. When they say: ...


3

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is correct. You can save a bit of memory by summing gradients as you go. Once you have calculated the gradients for one example for the weights of one layer, then you ...


3

You are correct, but requires final words: In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i.e. smooth error). On the other hand, in the Stochastic GD, we take one training sample to go one step towards the optimum, then repeat the latter for every ...


3

It is really simple. In gradient descent not using mini-batches, you feed your entire training set of data into the network and accumulate a cost function based on this full set of data. Then you use gradient descent to adjust the network weights to minimize the cost. Then you repeat this process until you get a satisfactory level of accuracy. For example, ...


2

The basic idea behind mini-batch training is rooted in the exploration / exploitation tradeoff in local search and optimization algorithms. You can view training of an ANN as a local search through the space of possible parameters. The most common search method is to move all the parameters in the direction that reduces error the most (gradient decent). ...


2

There is a trade-off between the: Memory capacity of computation device Quality of gradient approximation Generalization ability of the network Memory capacity I would say, that it is possible to process the whole dataset at once only for small enough dataset and image resolution (or any other measure of the data sample size - text sequence length, number ...


1

Technically, nothing prevents you from doing so. When you have mulitple losses, you may call .backward() at each term separately. However, I wonder, whether it makes sense to optimize each individual path as a separate objective, since if we have multiple of them - we would like to solve several tasks simultaneously. Probably, it could be beneficial as some ...


1

I am not sure if the process defined in the question is meaningful at all. If you mean to simply add the contribution of each $L$ without running the algorithm for the mini-batch, it makes no difference at all if you make one update or more; as the loss functions' contribution are simply added in the update. If on the other hand you mean to run the algorithm ...


1

In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have special dimension (usually corresponding to the 0 axis) which numerates the example in the batch. Look for example in the PyTorch documentation of Conv2d or Tensorflow documentation of Conv2d. The same is true for any Layer - Linear, MultiheadAttention, RNN. All samples from ...


1

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the explosion of the gradients magnitude and leads to more steady convergence. Adam is an adaptive optimizer with momentum and division by the weighted sum of gradients on ...


1

Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge and training will be very slow. With the advent of deep learning, training size is in millions and computation using all training examples is very impractical ...


Only top voted, non community-wiki answers of a minimum length are eligible