I have read Dense passage retrieval for Open Domain Question Answering, and in page 6 it talks about in-batch negative training, it states the following:
We find that using a similar configuration (7 gold negative passages), in-batch negative training improves the results substantially. The key difference between the two is whether the gold negative passages come from the same batch or from the whole training set. Effectively, in-batch negative training is an easy and memory-efficient way to reuse the negative examples already in the batch rather than creating new ones. It produces more pairs and thus increases the number of train- ing examples, which might contribute to the good model performance. As a result, accuracy consis- tently improves as the batch size grows.
I know that it is more efficient and hence we can increase the number of negatives by increasing the batch size (or we can use other techniques if we cannot do that), but in the paper they have the following table:
How having the same number of negatives (3rd & 4th rows) gave different results?
Is that something realted to the gradient or something, since we sample from the same examples that contribute to calculating the gradient in one step?