50
votes
What's the difference between model-free and model-based reinforcement learning?
What's the difference between model-free and model-based reinforcement learning?
In Reinforcement Learning, the terms "model-based" and "model-free" do not refer to the use of a ...
- 30.2k
35
votes
What's the difference between model-free and model-based reinforcement learning?
Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states $T$ ...
- 3,787
27
votes
What's the difference between model-free and model-based reinforcement learning?
In reinforcement learning (RL), there is an agent which interacts with an environment (in time steps). At each time step, the agent decides and executes an action, $a$, on an environment, and the ...
- 39.5k
9
votes
Is Q-learning a type of model-based RL?
Tabular Q-Learning does not explicitly create a model of the transition function. It does not generate any output that you can afterwards use as a function to predict what the next state s' will be ...
- 10.1k
8
votes
What's the difference between model-free and model-based reinforcement learning?
Although there are several good answers, I want to add this paragraph from Reinforcement Learning: An Introduction, page 303, for a more psychological view on the difference.
The distinction ...
- 787
5
votes
What's the difference between model-free and model-based reinforcement learning?
Model-Free RL
In Model-Free RL, the agent does not have access to a model of the environment. By environment I mean a function which predicts state transition and rewards.
As of the time of writing, ...
- 263
5
votes
How do temporal-difference and Monte Carlo methods work, if they do not have access to model?
The main idea is that you can estimate $V^\pi(s)$, the value of a state $s$ under a given policy $\pi$, even if you don't have a model of the environment, by visiting that state $s$ and following the ...
- 202
4
votes
Accepted
Into which subcategories can reinforcement learning be divided?
Your two suggestions are not mutually exclusive. If you go by this process, you'll have to do a "Cartesian product" of a bunch of different RL categorizations which would get out of hand. I ...
- 1,126
4
votes
Accepted
Are there RL algorithms that also try to predict the next state?
Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. ...
- 4,665
4
votes
Accepted
Is the state transition matrix known to the agents in a Markov decision processes?
In reinforcement learning (RL), there are some agents that need to know the state transition probabilities, and other agents that do not need to know. In addition, some agents may need to be able to ...
- 30.2k
3
votes
Accepted
Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse ...
- 1,251
3
votes
Is there any grid world dataset or generator for reinforcement learning?
Depending on your needs and the size of the project, you might be better off making a custom set of environments. If you'd rather not do that, though, you should take a look at OpenAI's CoinRun ...
- 2,028
3
votes
Accepted
How can the policy iteration algorithm be model-free if it uses the transition probabilities?
Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning.
See my answer ...
- 39.5k
2
votes
Is Q-learning a type of model-based RL?
In model-based learning, the learning agent utilizes a model that was previously learned to accomplish a task, whereas model-free RL doesn't use the environment to learn but simply relies on trial and ...
- 1,186
2
votes
What's the difference between model-free and model-based reinforcement learning?
According to OpenAI – Kinds of RL Algorithms, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called model-based methods, and those ...
- 313
2
votes
Accepted
Is the minimax algorithm model-based?
Minimax is a planning algorithm, and all planning algorithms need access to a model of the environment in order to look ahead or simulate possible future states and results.
Technically this does ...
- 30.2k
2
votes
Why are model-based methods more sample efficient than model-free methods?
In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one.
Main difference between those two is like ...
- 937
2
votes
Accepted
Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?
Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (...
- 30.2k
2
votes
Accepted
Have agents that "dream" been explored in Reinforcement Learning?
Yes, the concept of dreaming or imagining has already been explored in reinforcement learning.
For example, have a look at Metacontrol for Adaptive Imagination-Based Optimization (2017) by Jessica B. ...
- 39.5k
2
votes
Accepted
If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
However, if we can model the environment, why should we want to employ a model-free algorithm?
Depends what you mean by "model the environment". There are two kinds of model:
Distribution ...
- 30.2k
2
votes
Accepted
When we have multiple traces, do we average over traces or the total number of times we have visited that state?
For every visit MC you create a list for each state. Every time you enter a state you calculate the returns for the episode and append these returns to a list. Once you have done this for all the ...
- 4,665
2
votes
Are there RL algorithms that also try to predict the next state?
Check out Imagination-Augmented Agents paper - seems like it does what you are talking about. The agent itself is the standard A3C that you are familiar with. The novelty is the "imagination"...
- 2,426
2
votes
Model-based learning in continuous state and action spaces
You can use function approximation like neural networks to learn the whole environment, i.e. both the transition function, $p(s'\mid s, a)$, and the reward model, $r(s,a,s')$:
$$p(s',r\mid s,a)$$
In ...
- 2,443
2
votes
Accepted
In Q-Learning the Q-Table is not considered a model of the game?
The Q table is a useful summary of the underlying Markov Decision Process (MDP) model description of the environment and available choices. A Q table summarises expected results for a single policy - ...
- 30.2k
2
votes
Accepted
why learn an observation model when training latent space model in model based rl
The problem is the definition of what's $o_t$ and $s_t$:
$o_t$ is the (partially) observable part
$s_t$ is ideally a perfect model of the world
now, by definition, you don't have access directly to $...
- 1,308
1
vote
What is the difference between a distribution model and a sampling model in Reinforcement Learning?
I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily model something, unless, for example, you are ...
- 39.5k
1
vote
Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
I think there is an implicit notion of it in dynamic programming; say, if you have to make some sort of search over a subset of a state space and you are deciding whether to use BFS, breath first ...
- 121
1
vote
Accepted
What would be the reason behind using plots (such as box-plots or histograms) for ML development?
At a basic level, these kinds of low-dimensional plots where you look at one or two variables at a time can help to give you a sense of what types of relationships you might expect to see, such as ...
- 298
1
vote
Accepted
How does a model based agent learn the model?
If you already have some transition tuples then you can train a model to predict environment dynamics using these. However, you should be careful that your pre-gathered data is diverse enough to '...
- 4,665
1
vote
Are linear approximators better suited to some tasks compared to complex neural net functions?
This is just a case of supervised learning. You are trying to predict $s_{t+1}$ given $s_t$ and $a_t$, so the answer to your question depends on how complex your state dynamics are.
For example, if ...
- 4,665
Only top scored, non community-wiki answers of a minimum length are eligible