Questions tagged [model-based-methods]
For questions about model-based reinforcement learning methods (or algorithms). An example of a model-based algorithm is Dyna-Q, which estimates a model of the environment (i.e. the transition function of the associated Markov decision process).
41
questions
1
vote
0
answers
18
views
How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?
I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
- 19
1
vote
1
answer
36
views
why learn an observation model when training latent space model in model based rl
I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.
At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t ...
- 13
0
votes
3
answers
51
views
In Q-Learning the Q-Table is not considered a model of the game?
In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
1
vote
1
answer
56
views
Model-based learning in continuous state and action spaces
I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but ...
- 11
1
vote
1
answer
223
views
Model-based RL algorithms for continuous state space and finite action space
At the beginning, if I have a complete model $p(s' \mid s, a)$ (an assumed true model that describes the environment well enough) and the reward function $r(s,a,s')$. How can I exploit the model and ...
- 55
0
votes
0
answers
64
views
Looking for a reinforcement learning algorithm that deals well with a model-based, curiosity-driven approach for chess AI
I am a software engineer that meddled with machine learning (classifiers) during my thesis.
After being out of it for a while I decided I want to try and do a neural network project to learn from, ...
- 139
0
votes
0
answers
70
views
Reinforcement learning - confusion between model based and model free
I have have an environment with two models.
Model of the environment is stochastic. Given the price it returns the time when the next purchase will be made and how many items will be bought. Both of ...
0
votes
0
answers
17
views
Is model-based RL better suited for domain shift then model-free RL?
My intuition is that richer representations can be used for a larger number of downstream tasks and that model-based RL is more suited to produce such representations. Is there empirical work that ...
- 413
0
votes
0
answers
50
views
Example of games in reinforcement learning where no model is available? [duplicate]
I'm reading the Sutton & Barto's book "Reinforcement Learning: An Introduction" (2nd Edition), as the classes I took were a long time ago, and I'm struggling to understand this part (p. ...
1
vote
1
answer
470
views
In deep reinforcement learning, what is this model with state as input and value as output?
I was looking at this implementation for creating an agent for playing Tetris using DeepRL.
This model uses "a state based on the statistics of the board after a potential action. All predictions ...
- 165
2
votes
1
answer
72
views
If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is ...
- 23
1
vote
1
answer
133
views
Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
- 359
1
vote
2
answers
72
views
What would be the reason behind using plots (such as box-plots or histograms) for ML development?
I've been learning Python machine-learning using this project report and the guy who wrote it begins by visualizing his data using various statistical analysis methods: histograms, density plots, box ...
- 13
4
votes
1
answer
69
views
How does a model based agent learn the model?
I want to build model-based RL. I am wondering about the process of building the model.
If I already have data, from real experience:
$S_1, a \rightarrow R,S_2$
$S_2, a \rightarrow R,S_3$
Can I use ...
- 43
6
votes
2
answers
715
views
Are there RL algorithms that also try to predict the next state?
So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially.
If I remember correctly, all these ...
- 261
4
votes
1
answer
416
views
What is the difference between a distribution model and a sampling model in Reinforcement Learning?
The book from Sutton and Barto, Reinforcement Learning: An Introduction, define a model in Reinforcement Learning as
something that mimics the behavior of the environment, or more generally, that ...
- 141
1
vote
2
answers
248
views
Model-based RL for time series data
I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is.
To be concrete: I have $X(t)$ ...
- 11
2
votes
0
answers
102
views
What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?
In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods.
AlphaGo Deepmind RL model has beaten the best Go ...
- 201
0
votes
1
answer
123
views
When we have multiple traces, do we average over traces or the total number of times we have visited that state?
I am confused about the workings of the first- and every-visit MC.
My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?...
0
votes
0
answers
267
views
Using an LSTM for model-based RL in a POMDP
I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
- 1
1
vote
1
answer
404
views
Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
Here is David Silver's lecture on that. Look at 9:30 to 10:30.
He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.
But ...
- 155
4
votes
1
answer
824
views
Is the state transition matrix known to the agents in a Markov decision processes?
The question is more or less in the title.
A Markov decision process consists of a state space, a set of actions, the transition probabilities and the reward function. If I now take an agent's point ...
- 287
2
votes
1
answer
73
views
Are linear approximators better suited to some tasks compared to complex neural net functions?
Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
- 531
2
votes
2
answers
181
views
Have agents that "dream" been explored in Reinforcement Learning?
I was reading this article about the question "Why do we dream?" in which the author discusses dreams as a form of rehearsal for future threats, and presents it as an evolutive advantage. My ...
1
vote
1
answer
61
views
What is the expectation of an empirical model in model based RL?
In the paper - "Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems", on page 1083, on the 6th line from the bottom, the authors define ...
- 78
2
votes
1
answer
111
views
Into which subcategories can reinforcement learning be divided?
In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types.
So, into which subcategories can ...
- 23
3
votes
1
answer
63
views
Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?
In David Silver's 8th lecture he talks about model learning and says that learning $r$ from $s,a$ is a regression problem whereas learning $s'$ from $s,a$ is a kernel density estimation. His ...
- 4,665
1
vote
0
answers
62
views
Using a model-based method to build an accurate day trading environment model
There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects :
Value-based and policy-based
On-policy and off-policy
Model-free and ...
- 261
5
votes
3
answers
454
views
Isn't a simulation a great model for model-based reinforcement learning?
Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having ...
- 451
5
votes
2
answers
650
views
How can the policy iteration algorithm be model-free if it uses the transition probabilities?
I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given :
What I can't understand is ...
- 183
4
votes
1
answer
369
views
Is the minimax algorithm model-based?
Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a ...
- 613
2
votes
0
answers
59
views
Correlating two models to predict the output of one that corresponds to an output of the other
I am currently working on a problem and now got stuck to implement one of it's steps. This is a simple attempt to explain what I am currently facing, which is something that I am aiming to implement ...
- 141
4
votes
1
answer
2k
views
Why are model-based methods more sample efficient than model-free methods?
Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
- 451
2
votes
1
answer
876
views
How is Monte Carlo different from model-based methods?
I was going through an article where it is mentioned:
The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the ...
- 123
1
vote
0
answers
24
views
Architecture and Use of Different Algorithms for Health Goal Feedback
I wanted to get some opinions from the community for a certain problem that I will be approaching.
The problem is to provide feedback to a user based on a image of the upper male torso. The image ...
- 11
2
votes
1
answer
233
views
Is there any grid world dataset or generator for reinforcement learning?
I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
- 21
1
vote
2
answers
2k
views
How can we estimate the transition model and reward function?
In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model $p(s' \mid s, a)$ and the reward function $r(s, a)$, even ...
- 39.5k
4
votes
1
answer
170
views
How do temporal-difference and Monte Carlo methods work, if they do not have access to model?
In value iteration, we have a model of the environment's dynamics, i.e $p(s', r \mid s, a)$, which we use to update an estimate of the value function.
In the case of temporal-difference and Monte ...
- 41
1
vote
0
answers
100
views
Eligibility trace In Model-based Reinforcement Learning
In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace ...
- 481
3
votes
2
answers
2k
views
Is Q-learning a type of model-based RL?
Model-based RL creates a model of the transition function.
Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a ...
- 673
87
votes
6
answers
94k
views
What's the difference between model-free and model-based reinforcement learning?
What's the difference between model-free and model-based reinforcement learning?
It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
- 971