1
$\begingroup$

I read different articles and keep getting confused on this point. Not sure if the literature is giving mixed information or I'm interpreting it incorrectly.

So from reading articles my understanding (loosely) for the following terms are as follows:

Epoch: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.

Batch Size: Total number of training examples present in a single batch. In real life scenarios of utilising neural nets, the dataset needs to be as large as possible, for the network to learn better. So you can’t pass the entire dataset into the neural net at once (due to computation power limitation). So, you divide dataset into Number of Batches.

Iterations: Iterations is the number of batches needed to complete one epoch. We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.

So if all is correct, then my question is, at what point does the loss/cost function and the subsequent backprop processes take place (assuming from my understanding that backprop takes place straight after the loss/cost is calculated)? Does the cost/loss function gets calculated:

  1. At the end of each batch where the data samples in that batch have been forward-fed to the network (i.e. at each "Iteration, not each Epoch")? If so, then the loss/cost functions gets the average loss of all losses of all data samples in that batch, correct?

  2. At the end of each Epoch? Meaning all the data samples of all the batches are forward-fed first, before the a cost/loss function is calculated.

My understanding is that it's the first point, i.e. at the end of each batch (passed to the network), hence at each iteration (not Epoch). At least when it comes to SGD optimisation. My understanding is - the whole point is that you calculate loss/cost and backprop for each batch. That way you're not calculating the average loss of the entire data samples. Otherwise you would get a very universal minima value in the cost graph, rather than local minima with lower cost from each batch you train on separately. Once all iterations have taken place, then that would count as 1 Epoch. But then I was watching a YouTube video explaining Neural Nets, which mentioned that the cost/loss function is calculated at the end of each Epoch, which confused me. Any clarification would be really appreciated.

$\endgroup$
1
$\begingroup$

Epoch: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.

Batch Size: Total number of training examples present in a single batch. In real life scenarios of utilising neural nets, the dataset needs to be as large as possible, for the network to learn better. So you can’t pass the entire dataset into the neural net at once (due to computation power limitation). So, you divide dataset into Number of Batches.

Iterations: Iterations is the number of batches needed to complete one epoch. We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.

This is for the most part correct, except there are other reasons you would sometimes want to use batches (even if you could fit the whole thing in memory). One is that its less likely to overfit in a stochastic setting then in the full setting. Another is that it can achieve similar extrema with faster convergence.

Now regarding your question, yes you apply the gradient descent step at the end of each batch or desired batch (what I mean by desired batch, is if you want to use a batch-size of 24 but your device can only process 8, you may use gradient accumulation of 3 pseudo-batches to achieve an emulated batch of 24).

Though I do think its worth mentioning, your goal is still to find the global minimum. Even if your batch size is the entire dataset, that does not mean you will not fall into a local minimum, its actually in most cases still the most probable outcome

$\endgroup$
  • $\begingroup$ Thanks for the clarification. So my take aways from this (and correct me if I got this wrong), is that feeding the entire data set , in one go, to a neural network isn't recommended for the following reasons: 1) Today's computer's memory simply couldn't handle the huge datasets that neural nets need to train on, if all the dataset is fed in, in one go. 2) It will likely to achieve a local minimum (rather than a global one), since we're looking at an average gradient of loss function of the entire dataset, in one go (with respect to weights and baises). Hence SGD solves this by ... $\endgroup$ – Hazzaldo Aug 9 at 0:12
  • $\begingroup$ ... splitting data into batches, and therefore we will calculate the gradient of the loss for each batch (wrt weights & biases), rather the average of the entire dataset, and therefore would be able to find a much better minimum point of the loss function, that we're able to represent as the global minimum. Thus, we should always split our data into batches, and workout the loss and backprop for each batch. $\endgroup$ – Hazzaldo Aug 9 at 0:20
  • $\begingroup$ This would finally lead me onto the final takeaway: given what we discussed, the Loss and Backprop, therefore, takes place for each batch, NOT at the end of an Epoch. Once all batches have done their Loss calculations and Backprops, that would count as one Epoch. Again feel free to correct me if I got any of this wrong. $\endgroup$ – Hazzaldo Aug 9 at 0:21
  • $\begingroup$ One point I wasn't clear about, you mentioned: "Another is that it can achieve similar extrema with faster convergence". Sorry, could you just explain briefly what you mean by this? Also regarding this point: "if you want to use a batch-size of 24 but your device can only process 8, you may use gradient accumulation of 3 pseudo-batches to achieve an emulated batch of 24)", do you know if there's a good tutorial explaning on how this is done? Many thanks for your help and clear explanation. $\endgroup$ – Hazzaldo Aug 9 at 0:24
  • $\begingroup$ @Hazzaldo no. point 2 of takeaway is wrong. both strategies will lead to a minima no gaurantee on either. $\endgroup$ – mshlis Aug 9 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.