43 votes

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? In Reinforcement Learning, the terms "model-based" and "model-free" do not refer to the use of a ...
user avatar
  • 24.5k
29 votes

What's the difference between model-free and model-based reinforcement learning?

Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states $T$ ...
user avatar
21 votes

What's the difference between model-free and model-based reinforcement learning?

In reinforcement learning (RL), there is an agent which interacts with an environment (in time steps). At each time step, the agent decides and executes an action, $a$, on an environment, and the ...
user avatar
  • 34.5k
8 votes

Is Q-learning a type of model-based RL?

Tabular Q-Learning does not explicitly create a model of the transition function. It does not generate any output that you can afterwards use as a function to predict what the next state s' will be ...
user avatar
  • 9,459
7 votes

What's the difference between model-free and model-based reinforcement learning?

Although there are several good answers, I want to add this paragraph from Reinforcement Learning: An Introduction, page 303, for a more psychological view on the difference. The distinction ...
user avatar
5 votes

How do temporal-difference and Monte Carlo methods work, if they do not have access to model?

The main idea is that you can estimate $V^\pi(s)$, the value of a state $s$ under a given policy $\pi$, even if you don't have a model of the environment, by visiting that state $s$ and following the ...
user avatar
  • 202
4 votes
Accepted

Into which subcategories can reinforcement learning be divided?

Your two suggestions are not mutually exclusive. If you go by this process, you'll have to do a "Cartesian product" of a bunch of different RL categorizations which would get out of hand. I ...
user avatar
  • 981
4 votes

What's the difference between model-free and model-based reinforcement learning?

Model-Free RL In Model-Free RL, the agent does not have access to a model of the environment. By environment I mean a function which predicts state transition and rewards. As of the time of writing, ...
user avatar
  • 253
4 votes
Accepted

Are there RL algorithms that also try to predict the next state?

Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. ...
user avatar
4 votes
Accepted

Is the state transition matrix known to the agents in a Markov decision processes?

In reinforcement learning (RL), there are some agents that need to know the state transition probabilities, and other agents that do not need to know. In addition, some agents may need to be able to ...
user avatar
  • 24.5k
3 votes
Accepted

Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?

In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from Andrew Ng's Algorithms for Inverse ...
user avatar
  • 1,241
2 votes

Is Q-learning a type of model-based RL?

In model-based learning, the learning agent utilizes a model that was previously learned to accomplish a task, whereas model-free RL doesn't use the environment to learn but simply relies on trial and ...
user avatar
  • 1,146
2 votes

What's the difference between model-free and model-based reinforcement learning?

According to OpenAI – Kinds of RL Algorithms, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called model-based methods, and those ...
user avatar
2 votes
Accepted

How can the policy iteration algorithm be model-free if it uses the transition probabilities?

Everything you say in your post is correct, apart from the wrong assumption that policy iteration is model-free. PI is a model-based algorithm because of the reasons you're mentioning. See my answer ...
user avatar
  • 34.5k
2 votes
Accepted

Is the minimax algorithm model-based?

Minimax is a planning algorithm, and all planning algorithms need access to a model of the environment in order to look ahead or simulate possible future states and results. Technically this does ...
user avatar
  • 24.5k
2 votes

Why are model-based methods more sample efficient than model-free methods?

In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one. Main difference between those two is like ...
user avatar
  • 927
2 votes

Is there any grid world dataset or generator for reinforcement learning?

Depending on your needs and the size of the project, you might be better off making a custom set of environments. If you'd rather not do that, though, you should take a look at OpenAI's CoinRun ...
user avatar
2 votes
Accepted

Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?

Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (...
user avatar
  • 24.5k
2 votes
Accepted

Have agents that "dream" been explored in Reinforcement Learning?

Yes, the concept of dreaming or imagining has already been explored in reinforcement learning. For example, have a look at Metacontrol for Adaptive Imagination-Based Optimization (2017) by Jessica B. ...
user avatar
  • 34.5k
2 votes
Accepted

If we can model the environment, wouldn't be meaningless to use a model-free algorithm?

However, if we can model the environment, why should we want to employ a model-free algorithm? Depends what you mean by "model the environment". There are two kinds of model: Distribution ...
user avatar
  • 24.5k
2 votes
Accepted

When we have multiple traces, do we average over traces or the total number of times we have visited that state?

For every visit MC you create a list for each state. Every time you enter a state you calculate the returns for the episode and append these returns to a list. Once you have done this for all the ...
user avatar
2 votes

Are there RL algorithms that also try to predict the next state?

Check out Imagination-Augmented Agents paper - seems like it does what you are talking about. The agent itself is the standard A3C that you are familiar with. The novelty is the "imagination"...
user avatar
  • 1,833
1 vote

What is the difference between a distribution model and a sampling model in Reinforcement Learning?

I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily model something, unless, for example, you are ...
user avatar
  • 34.5k
1 vote

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?

I think there is an implicit notion of it in dynamic programming; say, if you have to make some sort of search over a subset of a state space and you are deciding whether to use BFS, breath first ...
user avatar
1 vote
Accepted

What would be the reason behind using plots (such as box-plots or histograms) for ML development?

At a basic level, these kinds of low-dimensional plots where you look at one or two variables at a time can help to give you a sense of what types of relationships you might expect to see, such as ...
user avatar
1 vote
Accepted

How does a model based agent learn the model?

If you already have some transition tuples then you can train a model to predict environment dynamics using these. However, you should be careful that your pre-gathered data is diverse enough to '...
user avatar
1 vote

Are linear approximators better suited to some tasks compared to complex neural net functions?

This is just a case of supervised learning. You are trying to predict $s_{t+1}$ given $s_t$ and $a_t$, so the answer to your question depends on how complex your state dynamics are. For example, if ...
user avatar
1 vote
Accepted

What is the expectation of an empirical model in model based RL?

If I understand your question correctly, the significance of this is due to the fact that $s'$ is random. In the RHS of the equation it is assumed that $V(\cdot)$ is known for each state, but the ...
user avatar
  • 981
1 vote

Isn't a simulation a great model for model-based reinforcement learning?

I will give one perspective on this from the domain of robotics. You are right that most RL agents are trained in simulation particularly for research papers, because it allows researchers to in ...
user avatar
1 vote

How can we estimate the transition model and reward function?

The original question about both the estimation of the transition model, often denoted as $T$, and the reward function, sometimes denoted as $R$, arose because I was thinking about the probability ...
user avatar
  • 34.5k

Only top scored, non community-wiki answers of a minimum length are eligible