Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using this approach for various corpora as well.

In the above approach, the concept of document does not play a role: $$pmi_{xy} = log\frac{p(x,y)} {p(x) p(y)}$$ where the probabilities are calculated across the whole corpus.

However, I am trying to generate document embeddings as well as word embeddings, so I used a document-based PMI approach, as outlined here : Word Tensors. In this approach, we consider the probability of the skipgram $(x,y)$ appears in document $d$: $$pmi_{xyd} = log \frac{p(x,y,d)}{p(x)p(y)p(d)}$$

I decompose the 3-mode tensor to get the document embedding along with the word embeddings.

Unfortunately, this approach does not work as well as the document-agnostic PMI approach and generates low-quality word embeddings. Further investigation reveals that the PMI information is not of good quality: even for a highly correlated skipgram (e.g., "husband" and "wife"), because we are only counting its frequency within each document (the $p(x,y,d)$ part) rather than across the whole corpus, and because single document is of limited length and provides limited contextual information, this highly correlated skipgram cannot get a high enough PMI value to distinguish itself from those skipgrams that are only weakly correlated.

So my question is: how do I improve the quality of the PMI information when we take documents into account and want to generate document embeddings? Is there research along this line? Thanks a lot to anyone who could point me to some relevant works!

(I originally asked this question on math overflow (link here), but did not get satisfactory response)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.