A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples.

This is difference than NCE Loss, which doesn't use a softmax at all, it uses a logistic binary classifier for the context/labels. In NLP, 'Negative Sampling' basically refers to the NCE-based approach.

More details here: https://www.tensorflow.org/extras/candidate_sampling.pdf.

I have tested both and they both give pretty much the same results. But in word embedding literature, they always use NCE loss, and never sampled softmax.

Is there any reason why this is? The sampled softmax seems like the more obvious solution to prevent applying a softmax to all the classes, so I imagine there must be some good reason for the NCE loss.



You must log in to answer this question.

Browse other questions tagged .