Suppose we want to perform importance sampling where we have trajectories from some behavioral policy $b$, but we want to perform off-policy evaluation. From these prior questions, I understand that importance sampling can be useful because we can easily sample from and evaluate the target policy $\pi$, while it is hard (or no longer possible) to sample from the original policy $b$. This is given by:
$$E_{x \sim \pi}[f(x)] \approx \frac{1}{n} \sum_{i=1}^n f(x_i) \frac{\pi(x_i)}{b(x_i)} $$
where the original trajectories $x_i$ were from $b$, but have now been reweighted as if they came from $\pi$.
However, if we no longer have access to sample from policy $b$, then how would we hope to easily evaluate the probability of some $x$ using policy $b$? Conversely, if we are able to easily evaluate $b(x_i)$, how come we can't sample from it? I'm especially interested in real-life examples where the probability distribution is not Poisson, Gaussian or some simple Bayes Net. For example, the probability distribution could be related to recognizing cats in images, dialogue generation or question answering.
