Questions tagged [on-policy-methods]
For questions related to the "on-policy" reinforcement learning algorithms.
34
questions
0
votes
1
answer
60
views
Understanding KL Stopping and KL Cutoff for the PPO algorithm
I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
- 608
1
vote
2
answers
44
views
Why does HER not work with on-policy RL algorithms?
I'm wondering because I don't appreciate what is wrong with just applying HER to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My ...
- 370
1
vote
1
answer
52
views
Are on-policy algorithms always better than off-policy ones?
I am studying RL and I have a question: Are on-policy algorithms always better than off-policy ones?
0
votes
0
answers
45
views
Are there papers that do an empirical investigation on DRL hyperparameters?
I am looking for papers that perform a study on DRL hyper-parameters. This paper does a fantastic job of describing the hyperparameters for on-policy algorithms. It would be great to get similar ...
- 608
0
votes
0
answers
129
views
Why are the actor and critic losses look weird in my PPO implementation?
I tried implementing the PPO algorithm on the Mujoco environment (InvertedDoublePendulum - v2). During the training, the actor-loss started from 10^(-1) magnitude and converged at 10^(-3) magnitude. ...
0
votes
1
answer
77
views
Can TRPO use replay buffers?
I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$.
Is it possible to ...
- 1
0
votes
1
answer
26
views
Using reinforcement learning for human-robot interaction [closed]
I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session.
I want to develop a ...
- 101
2
votes
0
answers
120
views
Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?
When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy).
I also saw that ACER has a higher time ...
- 115
2
votes
1
answer
259
views
Can off-policy algorithms benefit from the parallelization?
On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
- 341
1
vote
1
answer
118
views
Are the two policies in SARSA for choosing an action the same?
Here is the pseudocode for SARSA (which I took from here)
Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, ...
- 212
3
votes
0
answers
54
views
How to fix high variance of the returns on a 2d env?
I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution.
It is basically a 2d game where you have to move a small circle around the screen and try to avoid ...
- 31
2
votes
1
answer
540
views
How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?
I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy.
Can someone tell me how to code an $\epsilon$-soft policy?
I know how to ...
- 23
3
votes
1
answer
156
views
Is it possible to apply a particular exploration policy for the on-policy RL agents?
Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy ...
- 282
6
votes
1
answer
269
views
If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?
In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being
\begin{equation}
\mu(s) = \frac{\...
- 103
1
vote
0
answers
913
views
PPO in continuous control not working
I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
3
votes
1
answer
139
views
Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?
I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?
Knowing that, my main question concerns Q-learning and off-policy methods:
For me, there is something weird in ...
- 165
4
votes
1
answer
202
views
How should I generate datasets for a SARSA agent when the environment is not simple?
I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. ...
- 153
5
votes
1
answer
205
views
Why does off-policy learning outperform on-policy learning?
I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works.
I saw this in a book:
Q-learning is an off-policy learner. An off-...
- 293
4
votes
1
answer
151
views
What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?
I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that ...
- 95
1
vote
1
answer
299
views
Do we need multiple parallel environments to train in batches an on-policy algorithm?
When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
- 111
4
votes
1
answer
296
views
What is the difference between on-policy and off-policy for continuous environments?
I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space.
First, some preliminary questions: in this case, what is ...
- 331
1
vote
1
answer
478
views
Can we combine Off-Policy with On-Policy Algorithms?
On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms ...
- 451
2
votes
2
answers
1k
views
Why is DDPG an off-policy RL algorithm?
In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
- 150
1
vote
1
answer
806
views
What are the differences between 1-step SARSA and SARSA?
SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?
- 341
6
votes
1
answer
3k
views
Is Expected SARSA an off-policy or on-policy algorithm?
I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one.
Sutton and Barto's textbook describes Expected Sarsa thusly:
In these cliff walking results Expected Sarsa was ...
- 63
2
votes
0
answers
224
views
Action masking for on policy algorithm like PPO
I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
- 338
2
votes
1
answer
143
views
Could we update the policy network with previous trajectories using supervised learning?
I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is ...
- 451
2
votes
0
answers
73
views
Can we use imitation learning for on-policy algorithms?
Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
6
votes
1
answer
723
views
Convergence of semi-gradient TD(0) with non-linear function approximation
I am looking for a result that shows the convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (...
14
votes
1
answer
8k
views
What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?
In the context of RL, there is the notion of on-policy and off-policy algorithms. I understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the notion of ...
- 39.6k
2
votes
1
answer
222
views
Why is the actor-critic algorithm limited to using on-policy data?
Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
- 31
2
votes
1
answer
277
views
Do we need the transition probability function when calculating the importance sampling ratio?
I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused.
There is one thing I ...
4
votes
1
answer
2k
views
Why is GLIE Monte-Carlo control an on-policy control?
In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.
But why is it an on-policy control? The sampling follows a policy $\pi$ while ...
- 247
6
votes
2
answers
2k
views
What is the difference between on and off-policy deterministic actor-critic?
In the paper Deterministic Policy Gradient Algorithms, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic".
I don't know what's the difference ...
- 247