55
votes
What's the difference between model-free and model-based reinforcement learning?
What's the difference between model-free and model-based reinforcement learning?
In Reinforcement Learning, the terms "model-based" and "model-free" do not refer to the use of a ...
- 33.3k
36
votes
What's the difference between model-free and model-based reinforcement learning?
Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states $T$ ...
- 3,857
28
votes
What's the difference between model-free and model-based reinforcement learning?
In reinforcement learning (RL), there is an agent which interacts with an environment (in time steps). At each time step, the agent decides and executes an action, $a$, on an environment, and the ...
- 41.4k
9
votes
What's the difference between model-free and model-based reinforcement learning?
Although there are several good answers, I want to add this paragraph from Reinforcement Learning: An Introduction, page 303, for a more psychological view on the difference.
The distinction ...
- 797
6
votes
Accepted
Are model-free and off-policy algorithms the same?
In respect of RL, is model-free and off-policy the same thing, just different terminology?
No, they are entirely different terms, with the only thing they have in common is that they are both ways ...
- 33.3k
5
votes
What's the difference between model-free and model-based reinforcement learning?
Model-Free RL
In Model-Free RL, the agent does not have access to a model of the environment. By environment I mean a function which predicts state transition and rewards.
As of the time of writing, ...
- 273
5
votes
How do temporal-difference and Monte Carlo methods work, if they do not have access to model?
The main idea is that you can estimate $V^\pi(s)$, the value of a state $s$ under a given policy $\pi$, even if you don't have a model of the environment, by visiting that state $s$ and following the ...
- 202
4
votes
Accepted
Into which subcategories can reinforcement learning be divided?
Your two suggestions are not mutually exclusive. If you go by this process, you'll have to do a "Cartesian product" of a bunch of different RL categorizations which would get out of hand. I ...
- 1,136
4
votes
Accepted
How can the policy iteration algorithm be model-free if it uses the transition probabilities?
Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning.
See my answer ...
- 41.4k
3
votes
Accepted
How does one normalize observations in online reinforcement learning
Alternatively you can compute a running mean, $\mu_t$, and std, $\sigma_t$, of your online data $x_t$, and then standardize at each timestep, $t$:
$$\begin{align}
\mu_t &\leftarrow \mu_{t+1} + \...
- 3,186
3
votes
Accepted
Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse ...
- 1,291
3
votes
Accepted
Why are state-values alone not sufficient in determining a policy (without a model)?
why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values?
A policy function takes state as an argument and returns an action $a = \pi(s)$, or ...
- 33.3k
3
votes
What is the relation between Monte Carlo and model-free algorithms?
In Reinforcement Learning (RL), the use of the term Monte Carlo has been slightly adjusted by convention to refer to only a few specific things.
The more general use of "Monte Carlo" is for ...
- 33.3k
2
votes
What's the difference between model-free and model-based reinforcement learning?
According to OpenAI – Kinds of RL Algorithms, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called model-based methods, and those ...
- 313
2
votes
Accepted
If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
However, if we can model the environment, why should we want to employ a model-free algorithm?
Depends what you mean by "model the environment". There are two kinds of model:
Distribution ...
- 33.3k
2
votes
Accepted
Is the minimax algorithm model-based?
Minimax is a planning algorithm, and all planning algorithms need access to a model of the environment in order to look ahead or simulate possible future states and results.
Technically this does ...
- 33.3k
2
votes
Why are model-based methods more sample efficient than model-free methods?
In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one.
Main difference between those two is like ...
- 927
2
votes
Accepted
In Q-Learning the Q-Table is not considered a model of the game?
The Q table is a useful summary of the underlying Markov Decision Process (MDP) model description of the environment and available choices. A Q table summarises expected results for a single policy - ...
- 33.3k
1
vote
In Q-Learning the Q-Table is not considered a model of the game?
The key is that Q-learning does not use $p(s' \mid s, a)$ - the transition model - in its standard formulation. The Q-table isn't this model. It's not even an estimate of this model because it doesn't ...
- 41.4k
1
vote
In Q-Learning the Q-Table is not considered a model of the game?
No, a model in RL is considered something (you can even consider it as a black box) that knows how to transition from one state to the next one, so for example, if we are playing snake, if I give you ...
- 2,632
1
vote
How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?
Sutton and Barto explain it themselves in section 5.9. I post it with a bit of context. The equation you're looking for is 5.13.
- 348
1
vote
Why don't we bootstrap terminal state in n-step temporal difference prediction update equation?
Because the value of the terminal state is 0 by definition. There is no further reward to be obtained once you reach the terminal state.
- 348
1
vote
Accepted
In deep reinforcement learning, what is this model with state as input and value as output?
This is a variant of RL value-based approach using afterstate values. These are similar to action values, but have the following properties:
Afterstates treat an action as "choosing a next state&...
- 33.3k
1
vote
Accepted
How does policy evaluation work for continuous state space model-free approaches?
How does policy evaluation work for continuous state space model-free approaches? ... Let's say you use a DQN to find another policy, how does model-free policy evaluation work then?
Policy ...
- 1,703
1
vote
How can we estimate the transition model and reward function?
The original question about both the estimation of the transition model, often denoted as $T$, and the reward function, sometimes denoted as $R$, arose because I was thinking about the probability ...
- 41.4k
1
vote
Accepted
How can we estimate the transition model and reward function?
Given that model-based RL algorithms do not necessarily estimate or compute the transition model or reward function, in the case these are unknown, how can they be computed or estimated (so that they ...
- 33.3k
Only top scored, non community-wiki answers of a minimum length are eligible