43 votes

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? In Reinforcement Learning, the terms "model-based" and "model-free" do not refer to the use of a ...
user avatar
  • 24.5k
29 votes

What's the difference between model-free and model-based reinforcement learning?

Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states $T$ ...
user avatar
21 votes

What's the difference between model-free and model-based reinforcement learning?

In reinforcement learning (RL), there is an agent which interacts with an environment (in time steps). At each time step, the agent decides and executes an action, $a$, on an environment, and the ...
user avatar
  • 34.5k
7 votes

What's the difference between model-free and model-based reinforcement learning?

Although there are several good answers, I want to add this paragraph from Reinforcement Learning: An Introduction, page 303, for a more psychological view on the difference. The distinction ...
user avatar
5 votes

How do temporal-difference and Monte Carlo methods work, if they do not have access to model?

The main idea is that you can estimate $V^\pi(s)$, the value of a state $s$ under a given policy $\pi$, even if you don't have a model of the environment, by visiting that state $s$ and following the ...
user avatar
  • 202
4 votes

What's the difference between model-free and model-based reinforcement learning?

Model-Free RL In Model-Free RL, the agent does not have access to a model of the environment. By environment I mean a function which predicts state transition and rewards. As of the time of writing, ...
user avatar
  • 253
4 votes
Accepted

Into which subcategories can reinforcement learning be divided?

Your two suggestions are not mutually exclusive. If you go by this process, you'll have to do a "Cartesian product" of a bunch of different RL categorizations which would get out of hand. I ...
user avatar
  • 981
4 votes
Accepted

Are model-free and off-policy algorithms the same?

In respect of RL, is model-free and off-policy the same thing, just different terminology? No, they are entirely different terms, with the only thing they have in common is that they are both ways ...
user avatar
  • 24.5k
3 votes
Accepted

Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?

In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse ...
user avatar
  • 1,241
3 votes
Accepted

Why are state-values alone not sufficient in determining a policy (without a model)?

why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values? A policy function takes state as an argument and returns an action $a = \pi(s)$, or ...
user avatar
  • 24.5k
3 votes

What is the relation between Monte Carlo and model-free algorithms?

In Reinforcement Learning (RL), the use of the term Monte Carlo has been slightly adjusted by convention to refer to only a few specific things. The more general use of "Monte Carlo" is for ...
user avatar
  • 24.5k
2 votes

What's the difference between model-free and model-based reinforcement learning?

According to OpenAI – Kinds of RL Algorithms, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called model-based methods, and those ...
user avatar
2 votes
Accepted

If we can model the environment, wouldn't be meaningless to use a model-free algorithm?

However, if we can model the environment, why should we want to employ a model-free algorithm? Depends what you mean by "model the environment". There are two kinds of model: Distribution ...
user avatar
  • 24.5k
2 votes
Accepted

Is the minimax algorithm model-based?

Minimax is a planning algorithm, and all planning algorithms need access to a model of the environment in order to look ahead or simulate possible future states and results. Technically this does ...
user avatar
  • 24.5k
2 votes

Why are model-based methods more sample efficient than model-free methods?

In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one. Main difference between those two is like ...
user avatar
  • 927
2 votes
Accepted

How can the policy iteration algorithm be model-free if it uses the transition probabilities?

Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning. See my answer ...
user avatar
  • 34.5k
1 vote

How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

Sutton and Barto explain it themselves in section 5.9. I post it with a bit of context. The equation you're looking for is 5.13.
user avatar
  • 348
1 vote

Why don't we bootstrap terminal state in n-step temporal difference prediction update equation?

Because the value of the terminal state is 0 by definition. There is no further reward to be obtained once you reach the terminal state.
user avatar
  • 348
1 vote
Accepted

In deep reinforcement learning, what is this model with state as input and value as output?

This is a variant of RL value-based approach using afterstate values. These are similar to action values, but have the following properties: Afterstates treat an action as "choosing a next state&...
user avatar
  • 24.5k
1 vote
Accepted

How does policy evaluation work for continuous state space model-free approaches?

How does policy evaluation work for continuous state space model-free approaches? ... Let's say you use a DQN to find another policy, how does model-free policy evaluation work then? Policy ...
user avatar
  • 958
1 vote

How can we estimate the transition model and reward function?

The original question about both the estimation of the transition model, often denoted as $T$, and the reward function, sometimes denoted as $R$, arose because I was thinking about the probability ...
user avatar
  • 34.5k
1 vote
Accepted

How can we estimate the transition model and reward function?

Given that model-based RL algorithms do not necessarily estimate or compute the transition model or reward function, in the case these are unknown, how can they be computed or estimated (so that they ...
user avatar
  • 24.5k

Only top scored, non community-wiki answers of a minimum length are eligible