All Questions
Tagged with model-free-algorithms or model-free-methods
24 questions
1
vote
0
answers
31
views
How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?
I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
- 19
0
votes
3
answers
94
views
In Q-Learning the Q-Table is not considered a model of the game?
In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
2
votes
1
answer
326
views
How does one normalize observations in online reinforcement learning
I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its ...
- 670
0
votes
0
answers
76
views
Reinforcement learning - confusion between model based and model free
I have have an environment with two models.
Model of the environment is stochastic. Given the price it returns the time when the next purchase will be made and how many items will be bought. Both of ...
0
votes
0
answers
37
views
How to evaluate the performance of off-line & model-free reinforcement leaning?
I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
- 1
2
votes
1
answer
65
views
How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?
In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4:
the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
- 379
0
votes
1
answer
146
views
Why don't we bootstrap terminal state in n-step temporal difference prediction update equation?
In the algorithm below, when $\tau + n \geq T$, shouldn't the algorithm bootstrap with the value of the next state? For instance, when $T=5, \tau=3, \& \; n=2$, we don't bootstrap the sample ...
- 379
0
votes
0
answers
29
views
Why does one-step TD strengthen only the last action of the sequence of actions that led to the high reward, while n-step TD the last n actions?
In the caption of figure 7.4 (p. 147) of Sutton & Barto's book (2nd edition), it's written
The one-step method strengthens only the last action of the sequence of actions that led to the high ...
- 379
0
votes
0
answers
74
views
How does n-step Temporal Difference remove the notion of time-step?
How does n-step TD removes the notion of time-step as referenced in Sutton and Barto (2nd edition, Page 163) below?
Another way of looking at the benefits of n-step methods is that they free you from ...
- 379
1
vote
1
answer
519
views
In deep reinforcement learning, what is this model with state as input and value as output?
I was looking at this implementation for creating an agent for playing Tetris using DeepRL.
This model uses "a state based on the statistics of the board after a potential action. All predictions ...
- 165
2
votes
1
answer
86
views
If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is ...
- 23
2
votes
0
answers
220
views
What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?
In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods.
AlphaGo Deepmind RL model has beaten the best Go ...
- 201
1
vote
1
answer
481
views
Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
Here is David Silver's lecture on that. Look at 9:30 to 10:30.
He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.
But ...
- 155
4
votes
1
answer
517
views
Why are state-values alone not sufficient in determining a policy (without a model)?
"If a model is not available, then it is particularly useful to estimate action values (the
values of state-action pairs) rather than state values. With a model, state values alone are
sufficient ...
- 1,171
2
votes
1
answer
115
views
Into which subcategories can reinforcement learning be divided?
In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types.
So, into which subcategories can ...
- 23
6
votes
2
answers
874
views
How can the policy iteration algorithm be model-free if it uses the transition probabilities?
I'm actually trying to understand the policy iteration in the context of RL. I read an article presenting it and, at some point, a pseudo-code of the algorithm is given :
What I can't understand is ...
- 193
4
votes
1
answer
727
views
How does policy evaluation work for continuous state space model-free approaches?
How does policy evaluation work for continuous state space model-free approaches?
Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming ...
- 1,291
4
votes
1
answer
407
views
Is the minimax algorithm model-based?
Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a ...
- 633
5
votes
1
answer
2k
views
Are model-free and off-policy algorithms the same?
In respect of RL, is model-free and off-policy the same thing, just different terminology? If not, what are the differences? I've read that the policy can be thought of as 'the brain', or decision ...
- 633
5
votes
1
answer
2k
views
Why are model-based methods more sample efficient than model-free methods?
Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
- 471
2
votes
1
answer
1k
views
What is the relation between Monte Carlo and model-free algorithms?
Monte Carlo (MC) methods are methods that use some form of randomness or sampling. For example, we can use an MC method to approximate the area of a circle inside a square: we generate random 2D ...
- 41.4k
1
vote
2
answers
3k
views
How can we estimate the transition model and reward function?
In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model $p(s' \mid s, a)$ and the reward function $r(s, a)$, even ...
- 41.4k
5
votes
1
answer
184
views
How do temporal-difference and Monte Carlo methods work, if they do not have access to model?
In value iteration, we have a model of the environment's dynamics, i.e $p(s', r \mid s, a)$, which we use to update an estimate of the value function.
In the case of temporal-difference and Monte ...
- 51
92
votes
6
answers
103k
views
What's the difference between model-free and model-based reinforcement learning?
What's the difference between model-free and model-based reinforcement learning?
It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
- 1,021