3
$\begingroup$

The book from Sutton and Barto define a model in Reinforcement Learning as

"something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave." (Sutton-Barto, Reinforcement Learning: an Introduction).

In this answer, the answerer makes a distinction:

There are broadly two types of model:

  • A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

  • A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

The main difference is that in sampling models I only have a black box which, given a certain input $(s,a)$, generates an output, but I don't know anything about the probability distributions of the MDP. However, having a sampling model I can reconstruct (approximately) the probability distributions by running thousands of experiments (e.g. Monte Carlo Tree Search).

On the other hand, if I have a distribution model I can always sample from it.

I was wondering if

  1. what I wrote is correct;
  2. this distinction has been remarked in literature and where I can find a more in-depth discussion on the topic;
  3. someone has ever separated model-based algorithms which use a distribution model and model-based algorithms which use only a sampling model.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.