In-batch negative training Improves the results

Question

I have read Dense passage retrieval for Open Domain Question Answering, and in page 6 it talks about in-batch negative training, it states the following:

We find that using a similar configuration (7 gold negative passages), in-batch negative training improves the results substantially. The key difference between the two is whether the gold negative passages come from the same batch or from the whole training set. Effectively, in-batch negative training is an easy and memory-efficient way to reuse the negative examples already in the batch rather than creating new ones. It produces more pairs and thus increases the number of train- ing examples, which might contribute to the good model performance. As a result, accuracy consis- tently improves as the batch size grows.

I know that it is more efficient and hence we can increase the number of negatives by increasing the batch size (or we can use other techniques if we cannot do that), but in the paper they have the following table:

How having the same number of negatives (3rd & 4th rows) gave different results?

Is that something realted to the gradient or something, since we sample from the same examples that contribute to calculating the gradient in one step?

Answer 1

Yes, the In-batch negative training improves the results but not objectively, it is more helpful in reducing the confusion of semantics of various pattern which I guess tends to happen in deep-layers.

In the context of Blip-2, By including these hard negative samples in the training batch, the model is forced to learn to distinguish between positive and negative samples, which was helpful in Visual Knowledge Reasoning, but it didn't improve accuracy significantly.

For your last question regarding the link to gradient, yes it has to do with gradient, I guess.

        # sample way to generate hard negative samples
        with torch.no_grad():
            outputs = model(inputs)
            _, predicted = torch.max(outputs.data, 1)
            negatives = inputs[predicted != labels]

        #  hard negative samples with the positive samples
        inputs = torch.cat((inputs, negatives), dim=0)
        labels = torch.cat((labels, torch.zeros(negatives.shape[0], dtype=torch.long)), dim=0)

        # Zero the parameter gradients
        optimizer.zero_grad()