3
$\begingroup$

The book from Sutton and Barto, Reinforcement Learning: An Introduction, define a model in Reinforcement Learning as

something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.

In this answer, the answerer makes a distinction:

There are broadly two types of model:

  • A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

  • A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

The main difference is that in sampling models I only have a black box, which, given a certain input $(s,a)$, generates an output, but I don't know anything about the probability distributions of the MDP. However, having a sampling model, I can reconstruct (approximately) the probability distributions by running thousands of experiments (e.g. Monte Carlo Tree Search).

On the other hand, if I have a distribution model, I can always sample from it.

I was wondering if

  1. what I wrote is correct;

  2. this distinction has been remarked in literature and where I can find a more in-depth discussion on the topic;

  3. someone has ever separated model-based algorithms which use a distribution model and model-based algorithms which use only a sampling model.

$\endgroup$
1
$\begingroup$

I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily model something, unless, for example, you are first learning in simulation to later be able to act in the real-world or environment (in this sense, the simulation would be a model of the real environment, but this does not have to be the case, i.e. you may just want to act in the simulation (e.g. Atari games)), or, alternatively, when it's really a model of the MDP, but, in that case, you can just call it a model estimate.

So, you can call it a

  • sampling function, in case you sample e.g. from the experience replay
  • environment function, in case $r$ and $s'$ are returned by the environment,
  • model estimate, in case it's an estimate of $p(s' \mid s, a)$ (people may consider an experience replay a model estimate or, at least, information that can be used to build a model estimate)

The important thing to keep in mind is that, if you want to take an action $a$ in a certain state $s$, you need a function that returns you a reward $r$ and next state $s'$, if you want to do reinforcement learning.

I don't know if this distinction has been emphasized in the literature, but, as you noted, you can learn/estimate a (transition) model by exploring the world. I had asked a related question here a few years ago. You can also estimate the reward function, which is sometimes incorporated in the "model" of the environment, which, in this case, is denoted as $p(s', r \mid s, a)$ rather than just $p(s' \mid s, a)$, but these terms can be written as a function of each other.

People may also confuse this environment function with an exploratory policy, as they are, in a way, both used for exploration, but I think the concepts are distinct enough, as an exploratory policy is a way of deciding how to act given your current knowledge or ignorance: the exploratory policy can be viewed as a way of exercising/calling the environment function.

$\endgroup$
4
  • $\begingroup$ Thank you for the answer, @nbro. I also wouldn't a "sampling model" a "model", but it seems to me that this concept falls within the definition of a "model" given by Sutton and Barto, which I quoted in the question. I appreciate your comments. That said, you didn't fully answer to questions 2 and 3, that were the core of the question :) $\endgroup$
    – A. Pesare
    Jan 8 at 11:52
  • $\begingroup$ @A.Pesare Why does the "sampling model" fall into the category of "model"? A model in RL usually refers to a probability distribution that describes the dynamics of the environment, i.e. $p(s' \mid s, a)$. It only makes sense to call something a "sampling model" if that something is really a model or model estimate (as I call it) and if there's sampling involved (but you can also sample from $p(s' \mid s, a)$, so why not calling it also a sampling model?), if we want to be consistent with Sutton and Barto. $\endgroup$
    – nbro
    Jan 8 at 12:37
  • $\begingroup$ I've read many RL papers and I've never seen the word "sampling model" being used, as far as I remember. The closest to that is calling the experience replay technique a "model", but in its rawest form an ER is not a model, it's just a dataset, which you could use to build a model estimate. The word "sampling" is being used probably because we can "sample" from the "experience replay", which would be used to build the model, but this is a stretch. Again, the ER alone is not a model, it's a buffer/array/list/dataset of data, but it may induce a model. $\endgroup$
    – nbro
    Jan 8 at 12:40
  • $\begingroup$ If we follow the vague definition of a model that you quote "something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.", then yes you could consider an ER a model, but why call it a sampling model? As I said, yes, sometimes people may say that the ER is a model, but, in my view, ER is not directly a model, it's a dataset. It would be like calling a dataset in supervised learning a model too, but, in this case, it would be a different model. It doesn't make sense. The dataset is used to build the model. $\endgroup$
    – nbro
    Jan 8 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.