Questions tagged [model-based-methods]

For questions about model-based reinforcement learning methods (or algorithms). An example of a model-based algorithm is Dyna-Q, which estimates a model of the environment (i.e. the transition function of the associated Markov decision process).

Filter by
Sorted by
Tagged with
59
votes
6answers
50k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
6
votes
2answers
106 views

Are there RL algorithms that also try to predict the next state?

So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially. If I remember correctly, all these ...
5
votes
2answers
2k views

Is Q-learning a type of model-based RL?

Model-based RL creates a model of the transition function. Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a ...
5
votes
3answers
113 views

Isn't a simulation a great model for model-based reinforcement learning?

Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having ...
5
votes
0answers
69 views

Choosing Machine Learning Algorithm: Learning-Based Testing

This is my first project using machine learning so I'm looking for some guidance. I am extending a model-based testing (MBT) system to a learning-based testing system by integrating a machine learning ...
4
votes
1answer
136 views

How do temporal-difference and Monte Carlo methods work, if they do not have access to model?

In value iteration, we have a model of the environment's dynamics, i.e $p(s', r \mid s, a)$, which we use to update an estimate of the value function. In the case of temporal-difference and Monte ...
4
votes
1answer
245 views

Is the state transition matrix known to the agents in a Markov decision processes?

The question is more or less in the title. A Markov decision process consists of a state space, a set of actions, the transition probabilities and the reward function. If I now take an agent's point ...
4
votes
1answer
178 views

Is the minimax algorithm model-based?

Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a ...
4
votes
1answer
660 views

Why are model-based methods more sample efficient than model-free methods?

Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
4
votes
1answer
49 views

How does a model based agent learn the model?

I want to build model-based RL. I am wondering about the process of building the model. If I already have data, from real experience: $S_1, a \rightarrow R,S_2$ $S_2, a \rightarrow R,S_3$ Can I use ...
3
votes
1answer
50 views

Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?

In David Silver's 8th lecture he talks about model learning and says that learning $r$ from $s,a$ is a regression problem whereas learning $s'$ from $s,a$ is a kernel density estimation. His ...
3
votes
1answer
78 views

How can the policy iteration algorithm be model-free if it uses the transition probabilities?

I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : What I can't understand is ...
3
votes
0answers
35 views

Difference between a distribution model and a sampling environment in Reinforcement Learning

The book from Sutton and Barto define a model in Reinforcement Learning as "something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how ...
2
votes
1answer
92 views

Into which subcategories can reinforcement learning be divided?

In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types. So, into which subcategories can ...
2
votes
1answer
53 views

If we can model the environment, wouldn't be meaningless to use a model-free algorithm?

I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is ...
2
votes
2answers
118 views

Have agents that "dream" been explored in Reinforcement Learning?

I was reading this article about the question "Why do we dream?" in which the author discusses dreams as a form of rehearsal for future threats, and presents it as an evolutive advantage. My ...
2
votes
1answer
113 views

Is there any grid world dataset or generator for reinforcement learning?

I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
2
votes
1answer
64 views

Are linear approximators better suited to some tasks compared to complex neural net functions?

Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
2
votes
0answers
53 views

Correlating two models to predict the output of one that corresponds to an output of the other

I am currently working on a problem and now got stuck to implement one of it's steps. This is a simple attempt to explain what I am currently facing, which is something that I am aiming to implement ...
2
votes
1answer
296 views

How is Monte Carlo different from model-based methods?

I was going through an article where it is mentioned: The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the ...
2
votes
0answers
70 views

Eligibility trace In Model-based Reinforcement Learning

In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace ...
1
vote
1answer
122 views

Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?

Here is David Silver's lecture on that. Look at 9:30 to 10:30. He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used. But ...
1
vote
2answers
411 views

How can we estimate the transition model and reward function?

In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model (e.g. a probability distribution) and the reward function, even ...
1
vote
1answer
32 views

In deep reinforcement learning, what is this model with state as input and value as output?

I was looking at this implementation for creating an agent for playing Tetris using DeepRL. This model uses "a state based on the statistics of the board after a potential action. All predictions ...
1
vote
1answer
26 views

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
1
vote
2answers
63 views

What would be the reason behind using plots (such as box-plots or histograms) for ML development?

I've been learning Python machine-learning using this project report and the guy who wrote it begins by visualizing his data using various statistical analysis methods: histograms, density plots, box ...
1
vote
1answer
45 views

What is the expectation of an empirical model in model based RL?

In the paper - "Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems", on page 1083, on the 6th line from the bottom, the authors define ...
1
vote
1answer
87 views

Model-based RL for time series data

I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is. To be concrete: I have $X(t)$ ...
1
vote
0answers
70 views

What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?

In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ...
1
vote
0answers
52 views

Using a model-based method to build an accurate day trading environment model

There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects : Value-based and policy-based On-policy and off-policy Model-free and ...
1
vote
0answers
24 views

Architecture and Use of Different Algorithms for Health Goal Feedback

I wanted to get some opinions from the community for a certain problem that I will be approaching. The problem is to provide feedback to a user based on a image of the upper male torso. The image ...
0
votes
1answer
124 views

Can you use machine learning for binary data?

I am totally new to artificial intelligence and neural networks and have a broad question that I hope is appropriate to ask here. I am an ecologist working in animal movement and I want to use AI to ...
0
votes
1answer
49 views

When we have multiple traces, do we average over traces or the total number of times we have visited that state?

I am confused about the workings of the first- and every-visit MC. My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?...
0
votes
0answers
15 views

Why has PILCO not been included in Sutton & Barto?

PILCO is a model-based Reinforcement Learning method introduced in 2011 by Deisenroth and Rasmussen. As far as I know, it is still considered one of the most important RL method, especially for its ...
0
votes
0answers
19 views

Using an LSTM for model-based RL in a POMDP

I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...