Questions tagged [model-based-methods]

For questions about model-based reinforcement learning methods (or algorithms). An example of a model-based algorithm is Dyna-Q, which estimates a model of the environment (i.e. the transition function of the associated Markov decision process).

Filter by
Sorted by
Tagged with
1 vote
0 answers
26 views

How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?

I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
veerendra's user avatar
1 vote
1 answer
50 views

why learn an observation model when training latent space model in model based rl

I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley. At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t ...
platoDev's user avatar
0 votes
3 answers
60 views

In Q-Learning the Q-Table is not considered a model of the game?

In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
Raul Lapeira Herrero's user avatar
1 vote
1 answer
84 views

Model-based learning in continuous state and action spaces

I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but ...
hogger's user avatar
  • 11
1 vote
1 answer
287 views

Model-based RL algorithms for continuous state space and finite action space

At the beginning, if I have a complete model $p(s' \mid s, a)$ (an assumed true model that describes the environment well enough) and the reward function $r(s,a,s')$. How can I exploit the model and ...
k2pctdn's user avatar
  • 55
0 votes
0 answers
72 views

Reinforcement learning - confusion between model based and model free

I have have an environment with two models. Model of the environment is stochastic. Given the price it returns the time when the next purchase will be made and how many items will be bought. Both of ...
Evangelista01's user avatar
0 votes
0 answers
50 views

Example of games in reinforcement learning where no model is available? [duplicate]

I'm reading the Sutton & Barto's book "Reinforcement Learning: An Introduction" (2nd Edition), as the classes I took were a long time ago, and I'm struggling to understand this part (p. ...
FluidMechanics Potential Flows's user avatar
1 vote
1 answer
496 views

In deep reinforcement learning, what is this model with state as input and value as output?

I was looking at this implementation for creating an agent for playing Tetris using DeepRL. This model uses "a state based on the statistics of the board after a potential action. All predictions ...
JeanMi's user avatar
  • 165
2 votes
1 answer
78 views

If we can model the environment, wouldn't be meaningless to use a model-free algorithm?

I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is ...
Ayska's user avatar
  • 23
1 vote
1 answer
141 views

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?

Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
user529295's user avatar
1 vote
2 answers
72 views

What would be the reason behind using plots (such as box-plots or histograms) for ML development?

I've been learning Python machine-learning using this project report and the guy who wrote it begins by visualizing his data using various statistical analysis methods: histograms, density plots, box ...
MendelumS's user avatar
4 votes
1 answer
73 views

How does a model based agent learn the model?

I want to build model-based RL. I am wondering about the process of building the model. If I already have data, from real experience: $S_1, a \rightarrow R,S_2$ $S_2, a \rightarrow R,S_3$ Can I use ...
user46045's user avatar
6 votes
2 answers
846 views

Are there RL algorithms that also try to predict the next state?

So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially. If I remember correctly, all these ...
Ram Rachum's user avatar
4 votes
1 answer
472 views

What is the difference between a distribution model and a sampling model in Reinforcement Learning?

The book from Sutton and Barto, Reinforcement Learning: An Introduction, define a model in Reinforcement Learning as something that mimics the behavior of the environment, or more generally, that ...
A. Pesare's user avatar
  • 141
2 votes
2 answers
273 views

Model-based RL for time series data

I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is. To be concrete: I have $X(t)$ ...
Mariam 's user avatar
2 votes
0 answers
119 views

What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?

In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ...
user781486's user avatar
0 votes
1 answer
125 views

When we have multiple traces, do we average over traces or the total number of times we have visited that state?

I am confused about the workings of the first- and every-visit MC. My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?...
InvestingScientist's user avatar
0 votes
0 answers
293 views

Using an LSTM for model-based RL in a POMDP

I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
DECK's user avatar
  • 1
1 vote
1 answer
436 views

Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?

Here is David Silver's lecture on that. Look at 9:30 to 10:30. He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used. But ...
Aung Khant's user avatar
5 votes
1 answer
876 views

Is the state transition matrix known to the agents in a Markov decision processes?

The question is more or less in the title. A Markov decision process consists of a state space, a set of actions, the transition probabilities and the reward function. If I now take an agent's point ...
Felix P.'s user avatar
  • 297
2 votes
1 answer
73 views

Are linear approximators better suited to some tasks compared to complex neural net functions?

Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
mugoh's user avatar
  • 531
2 votes
2 answers
188 views

Have agents that "dream" been explored in Reinforcement Learning?

I was reading this article about the question "Why do we dream?" in which the author discusses dreams as a form of rehearsal for future threats, and presents it as an evolutive advantage. My ...
Bernardo Subercaseaux's user avatar
1 vote
1 answer
61 views

What is the expectation of an empirical model in model based RL?

In the paper - "Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems", on page 1083, on the 6th line from the bottom, the authors define ...
ijuneja's user avatar
  • 78
2 votes
1 answer
112 views

Into which subcategories can reinforcement learning be divided?

In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types. So, into which subcategories can ...
jackless's user avatar
3 votes
1 answer
63 views

Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?

In David Silver's 8th lecture he talks about model learning and says that learning $r$ from $s,a$ is a regression problem whereas learning $s'$ from $s,a$ is a kernel density estimation. His ...
David's user avatar
  • 4,790
1 vote
0 answers
63 views

Using a model-based method to build an accurate day trading environment model

There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects : Value-based and policy-based On-policy and off-policy Model-free and ...
jgauth's user avatar
  • 261
5 votes
3 answers
498 views

Isn't a simulation a great model for model-based reinforcement learning?

Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having ...
Ray Walker's user avatar
5 votes
2 answers
739 views

How can the policy iteration algorithm be model-free if it uses the transition probabilities?

I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given : What I can't understand is ...
Samuel Beaussant's user avatar
4 votes
1 answer
384 views

Is the minimax algorithm model-based?

Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a ...
mason7663's user avatar
  • 613
2 votes
0 answers
59 views

Correlating two models to predict the output of one that corresponds to an output of the other

I am currently working on a problem and now got stuck to implement one of it's steps. This is a simple attempt to explain what I am currently facing, which is something that I am aiming to implement ...
Marc Schwambach's user avatar
5 votes
1 answer
2k views

Why are model-based methods more sample efficient than model-free methods?

Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
Maybe's user avatar
  • 461
2 votes
1 answer
953 views

How is Monte Carlo different from model-based methods?

I was going through an article where it is mentioned: The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the ...
tausif's user avatar
  • 123
1 vote
0 answers
24 views

Architecture and Use of Different Algorithms for Health Goal Feedback

I wanted to get some opinions from the community for a certain problem that I will be approaching. The problem is to provide feedback to a user based on a image of the upper male torso. The image ...
Invic18's user avatar
  • 11
2 votes
1 answer
242 views

Is there any grid world dataset or generator for reinforcement learning?

I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
noor's user avatar
  • 21
1 vote
2 answers
2k views

How can we estimate the transition model and reward function?

In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model $p(s' \mid s, a)$ and the reward function $r(s, a)$, even ...
nbro's user avatar
  • 40.4k
4 votes
1 answer
176 views

How do temporal-difference and Monte Carlo methods work, if they do not have access to model?

In value iteration, we have a model of the environment's dynamics, i.e $p(s', r \mid s, a)$, which we use to update an estimate of the value function. In the case of temporal-difference and Monte ...
strongguy122's user avatar
1 vote
0 answers
112 views

Eligibility trace In Model-based Reinforcement Learning

In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace ...
Amin's user avatar
  • 481
3 votes
2 answers
2k views

Is Q-learning a type of model-based RL?

Model-based RL creates a model of the transition function. Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a ...
echo's user avatar
  • 693
91 votes
6 answers
97k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
mynameisvinn's user avatar
  • 1,011