In page 21 here, it states:

General Idea of Amortization: if same inference problem needs to be solved many times, can we parameterize a neural network to solve it?

Our case: for all $x^{(i)}$ we want to solve: $$ \min _{q(z)} \mathrm{KL}\left(q(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right. $$ Amortized formulation: $$ \min _{\phi} \sum \operatorname{KL}\left(q_{\phi}\left(z \mid x^{(i)}\right) \| p_{\theta}\left(z \mid x^{(i)}\right)\right) $$

One thing I am trying to wrap my mind around is $q(z)$ in the 1st formulation vs $q_{\phi}(z \mid x^{(i)})$ in the second.

Why is one a conditioned on $x^{(i)}$ and the other is not? I know in the 1st formulation, we are trying to find a different $q(z)$ for each datapoint.

I also recall that in VAEs,which uses amortized inference we consider $q(z)$ to be aggregated posterior, like

$$q_{\phi}(z)=\int q_{\phi}(x, z) d x \quad \text { Marginal of } q_{\phi}(x, z) \text { on } z$$

$$q_{\phi}(x, z) \equiv p_{\mathcal{D}}(x) q_{\phi}(z \mid x)$$

(formulas taken from here)


1 Answer 1


The reason is indeed the difference between traditional variational inference and the amortized variational inference.

In traditional VI, the objective is to solving the following problem for every datapoint $x^{(i)}$ $$\min _{q(z)} \mathrm{KL}\left(q(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$ Pay attention that if we write out explicitly the variational parameter $\phi$, it should depend on i-th datapoint (hence we call them local parameters), i.e. we are in fact solving N problems provided N is the number of datapoints. $$\min _{\phi_i} \mathrm{KL}\left(q_{\phi_i}(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$

while for amortized VI, we try to find a global variational parameter that serves for the inference of all the datapoints, i.e. $$\min _{\phi} \sum_{i=1}^N \operatorname{KL}\left(q_{\phi}\left(z \mid x^{(i)}\right) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$

This paper explains the difference with better notations.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .