# Difference between a distribution model and a sampling environment in Reinforcement Learning

The book from Sutton and Barto define a model in Reinforcement Learning as

"something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave." (Sutton-Barto, Reinforcement Learning: an Introduction).

In this answer, the answerer makes a distinction:

There are broadly two types of model:

• A distribution model which provides probabilities of all events. The most general function for this might be $$p(r,s'|s,a)$$ which is the probability of receiving reward $$r$$ and transitioning to state $$s'$$ given starting in state $$s$$ and taking action $$a$$.

• A sampling model which generates reward $$r$$ and next state $$s'$$ when given a current state $$s$$ and action $$a$$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

The main difference is that in sampling models I only have a black box which, given a certain input $$(s,a)$$, generates an output, but I don't know anything about the probability distributions of the MDP. However, having a sampling model I can reconstruct (approximately) the probability distributions by running thousands of experiments (e.g. Monte Carlo Tree Search).

On the other hand, if I have a distribution model I can always sample from it.

I was wondering if

1. what I wrote is correct;
2. this distinction has been remarked in literature and where I can find a more in-depth discussion on the topic;
3. someone has ever separated model-based algorithms which use a distribution model and model-based algorithms which use only a sampling model.