Questions tagged [model-based-methods]
For questions about model-based reinforcement learning methods (or algorithms). An example of a model-based algorithm is Dyna-Q, which estimates a model of the environment (i.e. the transition function of the associated Markov decision process).
37
questions
0
votes
0
answers
37
views
Looking for a reinforcement learning algorithm that deals well with a model-based, curiosity-driven approach for chess AI
I am a software engineer that meddled with machine learning (classifiers) during my thesis.
After being out of it for a while I decided I want to try and do a neural network project to learn from, ...
- 101
0
votes
0
answers
59
views
Reinforcement learning - confusion between model based and model free
I have have an environment with two models.
Model of the environment is stochastic. Given the price it returns the time when the next purchase will be made and how many items will be bought. Both of ...
0
votes
0
answers
12
views
Is model-based RL better suited for domain shift then model-free RL?
My intuition is that richer representations can be used for a larger number of downstream tasks and that model-based RL is more suited to produce such representations. Is there empirical work that ...
- 373
0
votes
0
answers
65
views
Devise a model/Goal Based Agent for simple Pacman Game
I have to program simple pacman game in figure below that consisting of 4*4 grid (not GUI based).
Explaination
The starting point of pacman is cell 0 and its goal is to consume/eat maximum food ...
0
votes
0
answers
46
views
Example of games in reinforcement learning where no model is available? [duplicate]
I'm reading the Sutton & Barto's book "Reinforcement Learning: An Introduction" (2nd Edition), as the classes I took were a long time ago, and I'm struggling to understand this part (p. ...
1
vote
1
answer
370
views
In deep reinforcement learning, what is this model with state as input and value as output?
I was looking at this implementation for creating an agent for playing Tetris using DeepRL.
This model uses "a state based on the statistics of the board after a potential action. All predictions ...
- 155
2
votes
1
answer
64
views
If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is ...
- 23
1
vote
1
answer
106
views
Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
Is there a notion of exploration-exploitation tradeoff in dynamic programming (or model-based RL)?
- 349
1
vote
2
answers
69
views
What would be the reason behind using plots (such as box-plots or histograms) for ML development?
I've been learning Python machine-learning using this project report and the guy who wrote it begins by visualizing his data using various statistical analysis methods: histograms, density plots, box ...
- 13
4
votes
1
answer
61
views
How does a model based agent learn the model?
I want to build model-based RL. I am wondering about the process of building the model.
If I already have data, from real experience:
$S_1, a \rightarrow R,S_2$
$S_2, a \rightarrow R,S_3$
Can I use ...
- 43
6
votes
2
answers
396
views
Are there RL algorithms that also try to predict the next state?
So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially.
If I remember correctly, all these ...
- 261
4
votes
1
answer
289
views
What is the difference between a distribution model and a sampling model in Reinforcement Learning?
The book from Sutton and Barto, Reinforcement Learning: An Introduction, define a model in Reinforcement Learning as
something that mimics the behavior of the environment, or more generally, that ...
- 141
1
vote
2
answers
198
views
Model-based RL for time series data
I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is.
To be concrete: I have $X(t)$ ...
- 11
2
votes
0
answers
93
views
What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?
In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods.
AlphaGo Deepmind RL model has beaten the best Go ...
- 201
0
votes
1
answer
103
views
When we have multiple traces, do we average over traces or the total number of times we have visited that state?
I am confused about the workings of the first- and every-visit MC.
My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?...
0
votes
0
answers
162
views
Using an LSTM for model-based RL in a POMDP
I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some ...
- 1
1
vote
1
answer
325
views
Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
Here is David Silver's lecture on that. Look at 9:30 to 10:30.
He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.
But ...
- 155
4
votes
1
answer
677
views
Is the state transition matrix known to the agents in a Markov decision processes?
The question is more or less in the title.
A Markov decision process consists of a state space, a set of actions, the transition probabilities and the reward function. If I now take an agent's point ...
- 287
2
votes
1
answer
72
views
Are linear approximators better suited to some tasks compared to complex neural net functions?
Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
- 511
2
votes
2
answers
171
views
Have agents that "dream" been explored in Reinforcement Learning?
I was reading this article about the question "Why do we dream?" in which the author discusses dreams as a form of rehearsal for future threats, and presents it as an evolutive advantage. My ...
1
vote
1
answer
57
views
What is the expectation of an empirical model in model based RL?
In the paper - "Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems", on page 1083, on the 6th line from the bottom, the authors define ...
- 78
2
votes
1
answer
105
views
Into which subcategories can reinforcement learning be divided?
In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types.
So, into which subcategories can ...
- 23
3
votes
1
answer
62
views
Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?
In David Silver's 8th lecture he talks about model learning and says that learning $r$ from $s,a$ is a regression problem whereas learning $s'$ from $s,a$ is a kernel density estimation. His ...
- 4,400
1
vote
0
answers
58
views
Using a model-based method to build an accurate day trading environment model
There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects :
Value-based and policy-based
On-policy and off-policy
Model-free and ...
- 161
5
votes
3
answers
321
views
Isn't a simulation a great model for model-based reinforcement learning?
Most reinforcement learning agents are trained in simulated environments. The goal is to maximize performance in (often) the same environment, preferably with a minimum amount of interactions. Having ...
- 451
5
votes
2
answers
433
views
How can the policy iteration algorithm be model-free if it uses the transition probabilities?
I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given :
What I can't understand is ...
- 183
4
votes
1
answer
301
views
Is the minimax algorithm model-based?
Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a ...
- 603
2
votes
0
answers
58
views
Correlating two models to predict the output of one that corresponds to an output of the other
I am currently working on a problem and now got stuck to implement one of it's steps. This is a simple attempt to explain what I am currently facing, which is something that I am aiming to implement ...
- 141
4
votes
1
answer
1k
views
Why are model-based methods more sample efficient than model-free methods?
Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
- 431
2
votes
1
answer
691
views
How is Monte Carlo different from model-based methods?
I was going through an article where it is mentioned:
The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the ...
- 123
1
vote
0
answers
24
views
Architecture and Use of Different Algorithms for Health Goal Feedback
I wanted to get some opinions from the community for a certain problem that I will be approaching.
The problem is to provide feedback to a user based on a image of the upper male torso. The image ...
- 11
2
votes
1
answer
196
views
Is there any grid world dataset or generator for reinforcement learning?
I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
- 21
1
vote
2
answers
1k
views
How can we estimate the transition model and reward function?
In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model $p(s' \mid s, a)$ and the reward function $r(s, a)$, even ...
- 37k
4
votes
1
answer
163
views
How do temporal-difference and Monte Carlo methods work, if they do not have access to model?
In value iteration, we have a model of the environment's dynamics, i.e $p(s', r \mid s, a)$, which we use to update an estimate of the value function.
In the case of temporal-difference and Monte ...
- 41
1
vote
0
answers
90
views
Eligibility trace In Model-based Reinforcement Learning
In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace ...
- 471
3
votes
2
answers
2k
views
Is Q-learning a type of model-based RL?
Model-based RL creates a model of the transition function.
Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a ...
- 663
77
votes
6
answers
83k
views
What's the difference between model-free and model-based reinforcement learning?
What's the difference between model-free and model-based reinforcement learning?
It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
- 871