The book from Sutton and Barto define a model in Reinforcement Learning as
"something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave." (Sutton-Barto, Reinforcement Learning: an Introduction).
In this answer, the answerer makes a distinction:
There are broadly two types of model:
A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.
A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.
The main difference is that in sampling models I only have a black box which, given a certain input $(s,a)$, generates an output, but I don't know anything about the probability distributions of the MDP. However, having a sampling model I can reconstruct (approximately) the probability distributions by running thousands of experiments (e.g. Monte Carlo Tree Search).
On the other hand, if I have a distribution model I can always sample from it.
I was wondering if
- what I wrote is correct;
- this distinction has been remarked in literature and where I can find a more in-depth discussion on the topic;
- someone has ever separated model-based algorithms which use a distribution model and model-based algorithms which use only a sampling model.