# What is $p(Z)$ and what happens to the variational posterior $q(Z;X)$ during data synthesis (after training)?

From my understanding of inference problems, we want to compute the posterior $$p(Z|X=D)$$, for some observed dataset $$D=(x^1, x^2,\dots,x^n)$$ of $$n$$ independent observations, in order to "update" our prior $$p(Z)$$ for further analysis/data generation. i.e. to sample new data $$x'\sim p(X)$$, we would sample $$z'\sim p(Z|X=D)$$, and then sample $$x'\sim p(X|z=z')$$.

However, in the VAE, we are generating (after training) new samples $$x'$$ by sampling $$z'\sim p(Z)=N(\bar{0},I)$$. If $$p(Z)$$ is simply our "prior belief" of the distribution of $$Z$$, prior to observing $$D$$, then why are new samples not generated by sampling $$z'\sim q(Z;X)$$, i.e. the variational approximation of the posterior?

1. One justification I can think of (but am not convinced by) is the $$D_{KL}(q(Z;X=x^i)||p(Z))$$ term in the ELBO objective, which ensures that the samplewise posterior (for every training sample) is close to $$p(Z)$$.

2. Another justification is that if we wanted to actually sample from $$q$$, we would need to determine $$q(Z;X=D)$$ (posterior probability of Z after observing entire $$D$$), which I think is difficult to compute. We would instead need to find some nice (with a functional form we can sample from) "average" distribution $$r(Z)$$ that is as close (min KL) to every samplewise posterior, which would basically result in $$r(Z)=p(Z)$$.

One way I can think of sampling a new $$z'$$ from the posterior would be to sample $$x\sim D$$ uniformly, and sample $$z'\sim q(Z;X=x)$$, and generate a new $$x'\sim p(X|Z=z')$$, which is essentially similar to one train step. To me this is like thinking of $$q(Z;X=D)$$ as a uniformly-weighted mixture of $$|D|$$ gaussians. This however, requires storage of $$O(|D|)$$ parameters (either the original data points, or the |D| latent means and stds).

Can someone please help me understand if I'm thinking about this correctly? What is the role of $$p(Z)$$? Why do we not use $$q(Z;X)\approx p(Z|X)$$ after training?