In the paper "Residual Energy-Based Models for Text Generation" (arXiv), on page 5, they write that equation 5 is an instance of importance sampling.

Equation 5 is:

$$ P(x_t \mid x_{<t}) = P_{LM}(x_t \mid x_{<t}) \, \frac{\mathbb{E}_{x'_{>t} \sim P_{LM}(\cdot \mid x_{\leq t})}[\exp(-E_\theta (x_{<t}, \, x_t, \, x'_{>t}))]}{\mathbb{E}_{x'_{\geq t} \sim P_{LM}(\cdot \mid x_{\leq t-1})}[\exp(-E_\theta (x_{<t}, \, x'_t, \, x'_{>t}))]} \ \ .$$

The goal is to approximate sampling from a distribution from which sampling is intractable $P_\theta(Y \mid X) = P_{LM}(Y \mid X) \, \frac{\exp(-E_\theta (X, Y))}{Z_\theta(X)}$, by sampling from $P_{LM}$, from which sampling is cheaper.

I understand that they are marginalizing over $>t$ in eq. 5, and I understand the basic idea of importance sampling to change $\mathbb{E}_{x \sim p}[f(x)]$ into $\mathbb{E}_{x \sim q}[f(x) \frac{p(x)}{q(x)}]$. However, eq. 5 is not a mean or aggregate, it is a probability.

What is happening? I don't see how eq. 5 fits in the importance sampling scheme (or a self-normalizing importance sampling scheme, link). Thanks in advance!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.