Assume 120 examples, a model makes 20 correct predictions and updates weight for the other 100. Should I count this epoch 100 iterations or 120?

Question

Per google's glossary, an iteration refers to

A single update of a model's weights during training ...

The following code comes from a github repo

def fit(self, x, y, verbose=False, seed=None):
    indices = np.arange(len(x))
    for i in range(self.n_epoch):
        n_iter = 0
        np.random.seed(seed)
        np.random.shuffle(indices)
        for idx in indices:
            if(self.predict(x[idx])!=y[idx]):
                self.update_weights(x[idx], y[idx], verbose)
            else:
                n_iter += 1
        if(n_iter==len(x)):
            print('model gets 100% train accuracy after {} epoch(s)'.format(i))
            break

Note that this model doesn't update weights for each single example, because when the model make a correct prediction for some example, it skips the example without updating weights.

In this kind of scenario where model makes a correct prediction for $i$th input $x_i$ and jump into next example $x_{i+1}$ without updating weights for $x_i$, does it count as an iteration?

Assume there are 120 training examples, in one epoch, the model makes 20 correct prediction and updates weight for the other 100. Should I count this epoch 100 iterations or 120 iterations?

Note: This question is NOT about coding. The code cited above works well. This question is about terminology. The code is just to illustrate the scenario in question.

Answer 1

This really depends on how you define "iteration". Here it's not so simple, given that the number of training steps per epoch varies based on the number of correct predictions, which would obviously change as you continue to train the model.

Generally I have found iterations refers to the number of times you run a batch through a network. This is the more reasonable definition, as when you are doing real-world machine learning, using batches of training examples allows you to utilize as much memory as possible, thus making the training much faster.

Using this definition with your example you would always have the same number of iterations as the batch size would be consistent, it's just inside the massive matrix that is your batch, you would have update values of 0 where the network correctly predicted.

It ultimately comes down to what's the best way to convey information to a reader? I think describing the number of iterations as variable based on network outputs is confusing and non-descriptive. In this case, if you don't want to use any batches, it is best to say you have 120 iterations per epoch. Saying you have up to 120 iterations is confusing. Instead, just specify that in some iterations the network may not be updated.

Answer 2

A quick google the keyword "definition of iteration in machine learning" gives us a lot of results. I would like to stick with this StackOverFlow question.

As your example, if we have 100 samples, let me assume batch size is 20, so the number of iteration is 5. If there is one iteration that both 20 samples are predicted correctly, this epoch should be counted which means one epoch you still have 5 iterations since the number of iterations is important in some situations such as control the learning rate while training (decay or cycle).

If you feel uncomfortable with the non-gradient / non-updating on that iteration, you can understand as your model's weight is updated with the gradient is 0.