1
$\begingroup$

In the paper "Residual Energy-Based Models for Text Generation" (arXiv), on page 5, they write that equation 5 is an instance of importance sampling.

Equation 5 is:

$$ P(x_t \mid x_{<t}) = P_{LM}(x_t \mid x_{<t}) \, \frac{\mathbb{E}_{x'_{>t} \sim P_{LM}(\cdot \mid x_{\leq t})}[\exp(-E_\theta (x_{<t}, \, x_t, \, x'_{>t}))]}{\mathbb{E}_{x'_{\geq t} \sim P_{LM}(\cdot \mid x_{\leq t-1})}[\exp(-E_\theta (x_{<t}, \, x'_t, \, x'_{>t}))]} \ \ .$$

The goal is to approximate sampling from a distribution from which sampling is intractable $P_\theta(Y \mid X) = P_{LM}(Y \mid X) \, \frac{\exp(-E_\theta (X, Y))}{Z_\theta(X)}$, by sampling from $P_{LM}$, from which sampling is cheaper.

I understand that they are marginalizing over $>t$ in eq. 5, and I understand the basic idea of importance sampling to change $\mathbb{E}_{x \sim p}[f(x)]$ into $\mathbb{E}_{x \sim q}[f(x) \frac{p(x)}{q(x)}]$. However, eq. 5 is not a mean or aggregate, it is a probability.

What is happening? I don't see how eq. 5 fits in the importance sampling scheme (or a self-normalizing importance sampling scheme, link). Thanks in advance!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.