A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples.

This is difference than NCE Loss, which doesn't use a softmax at all, it uses a logistic binary classifier for the context/labels. In NLP, 'Negative Sampling' basically refers to the NCE-based approach.

More details here: https://www.tensorflow.org/extras/candidate_sampling.pdf.

I have tested both and they both give pretty much the same results. But in word embedding literature, they always use NCE loss, and never sampled softmax.

Is there any reason why this is? The sampled softmax seems like the more obvious solution to prevent applying a softmax to all the classes, so I imagine there must be some good reason for the NCE loss.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.