0
$\begingroup$

Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using this approach for various corpora as well.

In the above approach, the concept of document does not play a role: $$pmi_{xy} = log\frac{p(x,y)} {p(x) p(y)}$$ where the probabilities are calculated across the whole corpus.

However, I am trying to generate document embeddings as well as word embeddings, so I used a document-based PMI approach, as outlined here : Word Tensors. In this approach, we consider the probability of the skipgram $(x,y)$ appears in document $d$: $$pmi_{xyd} = log \frac{p(x,y,d)}{p(x)p(y)p(d)}$$

I decompose the 3-mode tensor to get the document embedding along with the word embeddings.

Unfortunately, this approach does not work as well as the document-agnostic PMI approach and generates low-quality word embeddings. Further investigation reveals that the PMI information is not of good quality: even for a highly correlated skipgram (e.g., "husband" and "wife"), because we are only counting its frequency within each document (the $p(x,y,d)$ part) rather than across the whole corpus, and because single document is of limited length and provides limited contextual information, this highly correlated skipgram cannot get a high enough PMI value to distinguish itself from those skipgrams that are only weakly correlated.

So my question is: how do I improve the quality of the PMI information when we take documents into account and want to generate document embeddings? Is there research along this line? Thanks a lot to anyone who could point me to some relevant works!

(I originally asked this question on math overflow (link here), but did not get satisfactory response)

$\endgroup$

0

You must log in to answer this question.

Browse other questions tagged .