0
$\begingroup$

For nlp task like word2vec, we do negative sampling through the entire dataset

But in some cases like candidate generation in recommendation system, we do in batch negative sampling.

So my question are:

  1. What's the difference between two method?
  2. Did module like tensorflow has already realized in batch NEG and globally NEG?
$\endgroup$
1
  • $\begingroup$ Can you provide more details (e.g. links to references) that mention this "negative sampling" and this "batch negative sampling"? $\endgroup$
    – nbro
    Dec 11, 2022 at 12:08

1 Answer 1

0
$\begingroup$

In the original word2vec implementation, there was an initial pass on the entire dataset to build up the vocabulary. During training the negatives are randomly sampled from the entire vocabulary. The sampling strategy matters quite a bit. If we just sample every word with equal probability, we treat rare and frequent words alike. If we sample based on their frequency, we update the vectors of the frequent words far more often. The answer is to balance the two - subsample the frequent words. The authors report metrics with different such heuristics.

Recommender systems (using two tower DNN's) are usually trained using libraries like TF or Pytorch where training data is always batched. In this setting it's natural to get negatives from only within that batch. Fetching items from the entire dataset would be very very computationally inefficient. The same issue of oversampling frequent items occurs here too. Although we don't have global item frequency counts, sampling uniformly from every batch mimics sampling from the entire dataset weighted by frequency.There are approaches to correct for this.

So in summary

  1. Sampling randomly from every batch has the same effect as sampling from the entire dataset weighted by frequency
  2. Usually it helps to subsample frequent items or words.
  3. TF Recommenders allows specifying candidate sampling probability. Check the API
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .