From my understanding of inference problems, we want to compute the posterior $p(Z|X=D)$, for some observed dataset $D=(x^1, x^2,\dots,x^n)$ of $n$ independent observations, in order to "update" our prior $p(Z)$ for further analysis/data generation. i.e. to sample new data $x'\sim p(X)$, we would sample $z'\sim p(Z|X=D)$, and then sample $x'\sim p(X|z=z')$.
However, in the VAE, we are generating (after training) new samples $x'$ by sampling $z'\sim p(Z)=N(\bar{0},I)$. If $p(Z)$ is simply our "prior belief" of the distribution of $Z$, prior to observing $D$, then why are new samples not generated by sampling $z'\sim q(Z;X)$, i.e. the variational approximation of the posterior?
One justification I can think of (but am not convinced by) is the $D_{KL}(q(Z;X=x^i)||p(Z))$ term in the ELBO objective, which ensures that the samplewise posterior (for every training sample) is close to $p(Z)$.
Another justification is that if we wanted to actually sample from $q$, we would need to determine $q(Z;X=D)$ (posterior probability of Z after observing entire $D$), which I think is difficult to compute. We would instead need to find some nice (with a functional form we can sample from) "average" distribution $r(Z)$ that is as close (min KL) to every samplewise posterior, which would basically result in $r(Z)=p(Z)$.
One way I can think of sampling a new $z'$ from the posterior would be to sample $x\sim D$ uniformly, and sample $z'\sim q(Z;X=x)$, and generate a new $x'\sim p(X|Z=z')$, which is essentially similar to one train step. To me this is like thinking of $q(Z;X=D)$ as a uniformly-weighted mixture of $|D|$ gaussians. This however, requires storage of $O(|D|)$ parameters (either the original data points, or the |D| latent means and stds).
Can someone please help me understand if I'm thinking about this correctly? What is the role of $p(Z)$? Why do we not use $q(Z;X)\approx p(Z|X)$ after training?
