18
$\begingroup$

What's the difference between model-free and model-based reinforcement learning?

It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In that case, when would model-free learners be appropriate?

$\endgroup$
  • $\begingroup$ See also this answer: qr.ae/TUtHbv. $\endgroup$ – nbro Dec 9 '18 at 16:25
  • $\begingroup$ How do you mean that you could reframe a model-free learner as a model-based? $\endgroup$ – HelloGoodbye Feb 11 at 1:40
8
$\begingroup$

Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states $T$ and the reward function $R$. From this model, the agent has a reference and can plan accordingly.

However, it is not necessary to learn a model, and the agent can instead learn a policy directly using algorithms like Q-learning or policy gradient.

A simple check to see if an RL algorithm is model-based or model-free is:

If, after learning, the agent can make predictions about what the next state and reward will be before it takes each action, it's a model-based RL algorithm.

If it can't, then it’s a model-free algorithm.

$\endgroup$
  • 1
    $\begingroup$ in your words, "it is not necessary to learn a model", and my question is: why would anyone ever take a model-based approach? $\endgroup$ – vin Nov 10 '17 at 1:26
  • 3
    $\begingroup$ One big example I can think of is when you want an agent to learn about its surroundings without actually optimizing anything. This is part of the problem of continual learning, you need to build an internal model like "I hit walls when my distance sensor reads a wall is close" then that agent can generalize that information to multiple tasks if they arose. $\endgroup$ – Jaden Travnik Nov 10 '17 at 2:31
  • 1
    $\begingroup$ thanks @Jaden Travnik. i understand why it would be useful to learn a representation of the environment ("i hit walls when my distance reads a wall is close") without solving some task (eg navigating to the kitchen). but why would this be considered model-free RL, and not a vanilla supervised learning task? $\endgroup$ – vin Nov 11 '17 at 13:44
  • 1
    $\begingroup$ This wouldn't be supervised learning because there isn't any labeled data. The agent wouldn't have any idea what signals mean so couldn't tell a distance sensor from a thermometer. What the agent is learning is predictions of signals based on other signals, which is itself a model of its world. $\endgroup$ – Jaden Travnik Nov 11 '17 at 14:35
  • 1
    $\begingroup$ with a model-based approach, the agent learns to predict the next state, per your original explanation. it does so by learning <x, y>, where x is (s1, action) and y is (s2, reward). sorry if im misinterpreting, but isnt that supervised learning? $\endgroup$ – vin Nov 12 '17 at 13:30
4
$\begingroup$

What's the difference between model-free and model-based reinforcement learning?

In Reinforcement Learning, the terms "model-based" and "model-free" do not refer to the use of a neural network or other statistical learning model to predict values, or even to predict next state (although the latter may be used as part of a model-based algorithm and be called a "model" regardless of whether the algorithm is model-based or model-free).

Instead, the term refers strictly as to whether, whilst during learning or acting, the agent uses predictions of the environment response. The agent can use a single prediction from the model of next reward and next state (a sample), or it can ask the model for the expected next reward, or the full distribution of next states and next rewards. These predictions can be provided entirely outside of the learning agent - e.g. by computer code that understands the rules of a dice or board game. Or they can be learned by the agent, in which case they will be approximate.

Just because there is a model of the environment implemented, does not mean that a RL agent is "model-based". To qualify as "model-based", the learning algorithms have to explicitly reference the model:

  • Algorithms that purely sample from experience such as Monte Carlo Control, SARSA, Q-learning, Actor-Critic are "model free" RL algorithms. They rely on real samples from the environment and never use generated predictions of next state and next reward to alter behaviour (although they might sample from experience memory, which is close to being a model).

  • The archetypical model-based algorithms are Dynamic Programming (Policy Iteration and Value Iteration) or planning algorithms such as MCTS - these all use the model's predictions or distributions of next state and reward in order to calculate optimal actions. Specifically in Dynamic Programming, the model must provide state transition probabilities, and expected reward from any state, action pair. Note this is rarely a learned model.

  • Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions. In order to pick the best action, it needs to query a model that predicts what will happen on each action, and implement a policy like $\pi(s) = \text{argmax}_a \sum_{s',r} p(s',r|s,a)(r + v(s'))$ where $p(s',r|s,a)$ is the probability of receiving reward $r$ and next state $s'$ when taking action $a$ in state $s$. That function $p(s',r|s,a)$ is essentially the model.

The RL literature differentiates between "model" as a model of the environment for "model-based" and "model-free" learning, and use of statistical learners, such as neural networks.

In RL, neural networks are often employed to learn and generalise value functions, such as the Q value which predicts total return (sum of discounted rewards) given a state and action pair. Such a trained neural network is often called a "model" in e.g. supervised learning. However, in RL literature, you will see the term "function approximator" used for such a network to avoid ambiguity.

It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based.

I think here you are using the general understanding of the word "model" to include any structure that makes useful predictions. That would apply to e.g. table of Q values in SARSA.

However, as explained above, that's not how the term is used in RL. So although your understanding that RL builds useful internal representations is correct, you are not technically correct that this can be used to re-frame between "model-free" as "model-based", because those terms have a very specific meaning in RL.

In that case, when would model-free learners be appropriate?

Generally with current state of art in RL, if you don't have an accurate model provided as part of the problem definition, then model-free approaches are often superior.

There is lots of interest in agents that build predictive models of the environment, and doing so as a "side effect" (whilst still being a model-free algorithm) can still be useful - it may regularise a neural network or help discover key predictive features that can also be used in policy or value networks. However, model-based agents that learn their own models for planning have a problem that inaccuracy in these models can cause instability (the inaccuracies multiply the further into the future the agent looks). Some promising inroads are being made using imagination-based agents and/or mechanisms for deciding when and how much to trust the learned model during planning.

Right now (in 2018), if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as DQN or A3C. That may change as the field is moving fast and new more complex architectures could well be the norm in a few years.

$\endgroup$
1
$\begingroup$

According to OpenAI – Kinds of RL Algorithms, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called model-based methods, and those that don’t are called model-free. This model can either have been given the agent or learned by the agent.

Using a model allows the agent to plan by thinking ahead, seeing what would happen for a range of possible choices, and explicitly deciding between its options. This may be useful when faced with problems that require more long-term thinking. One way to perform planning is by using some kind of tree search, for example Monte Carlo tree search (MCTS), or—which I suspect could also be used—variants of the rapidly exploring random tree (RRT). See e.g. Agents that imagine and plan.

The agent can then distill the results from planning ahead into a learned policy – this is known as expert iteration.

A model can also be used to create a simulated, or "imagined," environment in which the state is updated by using the model, and make the agent learn inside of that environment, such as in World Models.

In many real-world scenarios, the ground-truth model of the environment is not available to the agent. If an agent wants to use a model in this case, it has to learn the model, which can be challenging for several reasons.

There are however cases in which the agent uses a model that is already known and consequently doesn't have to learn the model, such as in AlphaZero, where the model comes in form of the rules of the game.

$\endgroup$
0
$\begingroup$

In RL, the environment which the agent interacts with, the actions the agent can take in that same environment and the possible rewards emitted by the environment are all together (often) represented by a Markov decision process (MDP) or partially observable Markov decision process (POMDP). For simplicity, let's just consider the MDP. The MDP defines the probabilities of transitioning from one state of the environment to another, the probabilities of taking certain actions from certain states and the probabilities of obtaining certain "rewards" after the agent has moved from one state to the other (after having taken a certain action). The MDP is a probabilistic or stochastic model. More concretely, for example, if the agent is in a certain state, the agent may not be sure which state to move in or which action to take in the next time step; these uncertainties can be captured by the probabilistic nature of the MDP.

A model-based algorithm is one that uses these probabilities of the MDP. A model-free algorithm does not use them.

The best way to understand if an algorithm is model-based or model-free is to look at it and see if uses these probabilities or not.

For instance, let's look at the main update rule in the Q-learning algorithm:

$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha (R_{t+1} + \gamma \max_{a}Q(S_{t+1}, a) - Q(S_t, A_t))$$

As we can see, this update rule does not use any probabilities defined by the MDP. Note: $R_{t+1}$ is just the reward that is obtained at the next time step (after taking the action), but it is not necessarily known beforehand. So, Q-learning is a model-free algorithm.

Now, let's look at the main update rule of the policy improvement algorithm:

$$Q(s,a) \leftarrow \sum_{s' \in \mathcal{S}, r\in\mathcal{R}}p(s',r|s,a)(r+\gamma V(s'))$$

We can immediately observe it uses $p(s',r|s,a)$, a probability defined by the MDP model. So, policy iteration (a dynamic programming algorithm), which uses the policy improvement algorithm, is a model-based algorithm.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.