After reading through this thread and some other resources online, I still do not understand the role of epochs in training a neural network. I understand that one epoch is one iteration through the entire data set. But I don't understand what happens when we finish one epoch and start the second, i.e., what is happening such that the error is reduced after each epoch?

Assuming we have an outer loop over epochs, and an inner loop for the gradient descent iteration. What happens is the following, at least to my understanding:

For the first epoch, we take some random initial parameters and perform, say, 1000 gradient descent steps until we have found a local minima where the loss is minimised. So we have landed in a local minima, but it could be a bad one and we want to find a better one, preferably even the global minima. However, this is what I was thinking about:

The only way we can escape a bad local minima is to start a new gradient descent search with different initial parameters, but it seems like starting a new epoch is not just taking different parameters and repeating the process. According to the loss plotted over epochs, the loss is decreasing after each iteration, so it seems like the parameters are not chosen randomly, instead it looks like there is some knowledge used from the previous epoch. First, I assumed that a new epoch just means starting a new search over the loss landscape, this time with another set of random initial values, such that we end up in a different local minima. However, it is not guaranteed that this local minima is a better one, so we could also land in a worse local minima.

But if I look at graphs where the loss is plotted over epochs, it almost always decreases quite smoothly with each epoch, indicating that every epoch does indeed lead to a better minima. This is where I get confused. How is this possible? What is happening at the start of a new epoch? In order to decrease the error, we must have knowledge from the previous iteration, which means that we do not just simply start with a different random set of initial parameters. So, how does a new epoch iteration contribute to a better result? What is happening that the loss is decreasing without ever rising due to worse local minima?

------------ EDIT ------------

I want to add this simple example: consider a simple linear network, no activation function. We have 2 inputs, each multiplied by a weight to generate a scalar output, i.e. the first step would look like:

x1[0]*w1 + x2[1]*w2 = out

where x1[0] contains the initial value and x2 is a different, known value. We want to find the weights w1 and w2 which minimise the loss between the output and the ground truth. In this case, I will work with one batch only, which contains the 2 inputs. The goal is to approximate a function linearly, given a known initial value. Then, I would run the training loop as follows

net = Net(2,1)
criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(),lr=0.01)

for m in range(1,11):
    inputs = torch.Tensor(x[m-1], x[m]])
    for iteration in range(5000):
        out = net(inputs)
        loss = criterion(out, truth)

This is just as snapshot of the loop, but basically the idea here is to find the appropriate weights for each point m of the function I want to approximate, by repeating the gradient descent search until we have found the local minima, and then proceed to the next point m+1 and so on. Using this example, how and why would a third loop over epochs decrease the error?


1 Answer 1


I am not expert on optimization, but I can share with you my knowledge of the topic.

I think that the source of your confusion is that you assumed that, after the first epoch, we have reached a local minimum. This is not necessarily true, although we might be on our way to get there, so that's why we usually have more than one epoch, and this should also explain why the loss decreases over epochs.

Moreover, as you correctly realised, we don't start the next epoch with new random weights. We start it with the weights that we found in the previous epoch.

Lastly, we might train the neural network with mini-batch (stochastic) gradient descent, which means that we split the original training dataset into $M = \lceil N / B \rceil$ batches, where $N$ is the size of the original dataset and $B$ is the batch size (typical numbers are 32 or 64). So, if you have, say, $N=10000$ and $B = 32$, then we have $M = 312$ mini-batches; so, for each epoch, you will perform 312 gradient steps, one for each mini-batch.

Now, although we use the same training data in different epochs, there are at least 2-3 reasons why the result of GD at the end of these epochs is different

  1. at the beginning of each epoch, we may randomly shuffle the original training dataset before splitting it into mini-batches
  2. even if that's not the case, we may feed the mini-batches in different order to the neural network
  3. we start the epochs with different weights (e.g. in the first epoch, we start with random weights, in the second, we start with the weights found in the first epoch, and so on); think of a loss landscape with multiple minima (figure 1 here): the result of the optimization process depends on the starting point on this loss landscape
  • 1
    $\begingroup$ but why would I terminate the inner loop which is the optimisation loop when I have not found a stationary point? I do as many iterations as needed until the gradients vanish, so I know I am at a local min. And if the new epoch starts, with the parameters from the last epoch, well we are at the same point again, so how do we escape from there? Or, is the loop over epochs what I call here the optimisation loop? For simplicity, lets assume I do not have multiple batches, but one batch with all the data $\endgroup$
    – tanasr
    Commented Apr 8, 2022 at 16:56
  • $\begingroup$ I guess the answer to your question is in the 1st and 2nd point (regarding GD) in my answer, i.e. at the beginning of every epoch, you may randomly shuffle the training dataset before splitting it into mini-batches or, alternatively, you may feed the model with another (probably random) order of the mini-batches (wrt the previous epoch). $\endgroup$
    – nbro
    Commented Apr 8, 2022 at 17:00
  • $\begingroup$ You could do what you say, i.e. not have epochs, but, if after you've gone through all your training data (with the mini-batches), you shuffle the training data again, conceptually, it makes sense to highlight that point in time (i.e. the beginning of another epoch). Right now, I don't remember whether libraries like Keras or PyTorch do exactly this (i.e. whether they shuffle the training dataset at the beginning of every epoch before splitting). You may want to look at that ;) $\endgroup$
    – nbro
    Commented Apr 8, 2022 at 17:00
  • $\begingroup$ In any case, this is just a conceptual distinction, i.e. even if you shuffle the training data at some point, you could not have the epochs and just have "GD updates". One library (which I know of) that does something similar, but in the context of reinforcement learning, is stable-baselines. It doesn't have episodes (which would be the rough equivalent of epochs in supervised learning) but just "training steps". $\endgroup$
    – nbro
    Commented Apr 8, 2022 at 17:06
  • 1
    $\begingroup$ thank you for your thoughtful explanations! after going through the documentation of PyTorch and some introductory videos I figured out where the confusion comes from: I approached the training of a neural network from an optimisation methods point of view. What you explained makes totally sense, however, I was picturing a very simple linear regression problem with 2 inputs and one output. For anyone having the same confusion, I will post an answer to my own question where I have more characters left:) $\endgroup$
    – tanasr
    Commented Apr 9, 2022 at 13:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .