2
$\begingroup$

I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is about the policy. However, if we can model the environment, why should we want to employ a model-free algorithm? Isn't it better to have a model and expectation about the next reward and state? If you have a better understanding of all these, can you explain them to me as well?

$\endgroup$
2
  • 1
    $\begingroup$ I am not sure what you mean by "It is about the policy"? Could you please clarify? $\endgroup$ Aug 4 at 16:26
  • $\begingroup$ I mean, even if we have a model, the main reason for having this model is to find the optimal policy. (If I am right!). $\endgroup$
    – Ayska
    Aug 5 at 12:37
2
$\begingroup$

However, if we can model the environment, why should we want to employ a model-free algorithm?

Depends what you mean by "model the environment". There are two kinds of model:

  • Distribution model, which provides full access to a function like $p(r,s'|s,a)$, the probability of observing reward $r$ and next state $s'$ given starting state $s$ and taking action $a$.

  • Sample model, which can process a single step within the environment on demand. That should provide access to a function like $\text{step}(s,a)$ that returns a single $(r, s')$ pair based on running the environment forward from $(s,a)$.

Which kind of model is available makes a difference to which kinds of model-based reinforcement-learning approaches will work. For instance a distribution model is required for dynamic programming, but MCTS may only need a sample model.

A distribution model can be converted easily to a sample model, but the other way around is only possible approximately (by taking lots of samples).

Isn't it better to have a model and expectation about the next reward and state?

All things being equal, then yes having access to and using a model allows for better estimates, and opens up possibilities for various types of planning algorithm.

However, even if you can build a model of an environment, there are costs:

  • Writing a distribution model can be hard theoretically - it is often far easier to simulate an environment for a single step forward than calculate probability distributions for all outcomes of the same step.

  • The need for random access to arbitrary states can be expensive. A simulator that only needs to run forward from a single reset to a start state may run much more efficiently than one that needs to be set to multiple places in a tree of possibilities arbitrarily.

These costs are different for different environments. Setting a game like chess or go to arbitrary states is relatively cheap compared to a modern computer game or something that simulates a real-world physical system with lots of detail. Partially-observable states can be a major problem in both cases, since any random access model would need to account for all possible "true" states that lie behind whatever the agent is processing.

So basically, at different levels of environment complexity, either the development cost of making a full model, or the evaluation cost of setting arbitrary states can make it more effective to use a model-free approach.

Agents that learn to build their own approximate models of the environment are of great interest, as these potentially address the issue by getting the best of both world. Or possibly they could be tunable to use model-free vs model-based approaches in the most efficient manner depending on the relative costs and accuracy of each approach.

$\endgroup$
9
  • $\begingroup$ I'm not sure I understand why you differentiate between what you call a "distribution model" and a "sampling model". You always need something that returns you the next state $s'$ and reward $r$ for every state $s$ and action $a$ for any RL algorithm; so, if I understand correctly your terminology, you always need a "sampling model" anyway. It seems to me that a sampling model is just the distribution model, but without knowing the probabilities, i.e. just a way to sample from $p.$ $\endgroup$
    – nbro
    Aug 4 at 17:30
  • $\begingroup$ I wouldn't call it a sampling model, but step/sampling function (as you do) to differentiate it from what we usually call "model" in RL (not sure if "sampling model" is used in the literature, I don't know). The step function may be stochastic, but we may not know exact how it computes. Or your "sampling model" would be kind of a simulation of the real environment? $\endgroup$
    – nbro
    Aug 4 at 17:36
  • $\begingroup$ @nbro: Sample model and distribution model are terms used in e.g. Sutton & Barto. A distribution model can produce a sample model, but not the other way around (except via some statistical learning algorithm). Technically the book uses "sample model", not "sampling model", so I'll change my answer to match $\endgroup$ Aug 4 at 17:49
  • $\begingroup$ Ok. It's been a while since I read that book, and probably that term is only used in some chapters and maybe in the 2nd ed (I will check that). $\endgroup$
    – nbro
    Aug 4 at 17:55
  • $\begingroup$ @nbro: You will not need a sample model if you have access to the real environment. As some real environments are virtual, implemented in code, then sometimes that line is blurred. Anyway, with a real environment, the step function is not calling a model, it is calling the real instance of the thing. $\endgroup$ Aug 4 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.