Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [on-policy-methods]

For questions related to the "on-policy" reinforcement learning algorithms.

Filter by
Sorted by
Tagged with
0 votes
1 answer
56 views

It is not clear why SARSA is on-policy but Q-learning off-policy

Here are SARSA and Q-learning from Sutton & Barto. In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if ...
DSPinfinity's user avatar
0 votes
1 answer
40 views

Policy in on-policy algorithms and experience replay

For SARSA algorithm, assuming that we initialize all $Q(s,a)$ to $0$, then in the first iteration, all actions are the best actions as $Q$ values are the same ($0$). So the behavior policy in this ...
k2pctdn's user avatar
  • 55
0 votes
1 answer
239 views

Understanding KL Stopping and KL Cutoff for the PPO algorithm

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
desert_ranger's user avatar
1 vote
2 answers
57 views

Why does HER not work with on-policy RL algorithms?

I'm wondering because I don't appreciate what is wrong with just applying HER to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My ...
profPlum's user avatar
  • 424
1 vote
1 answer
68 views

Are on-policy algorithms always better than off-policy ones?

I am studying RL and I have a question: Are on-policy algorithms always better than off-policy ones?
Samvel Safaryan's user avatar
0 votes
1 answer
103 views

Can TRPO use replay buffers?

I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$. Is it possible to ...
Wj210's user avatar
  • 1
0 votes
1 answer
27 views

Using reinforcement learning for human-robot interaction [closed]

I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session. I want to develop a ...
caaax's user avatar
  • 101
2 votes
0 answers
140 views

Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?

When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy). I also saw that ACER has a higher time ...
kitaird's user avatar
  • 115
2 votes
1 answer
349 views

Can off-policy algorithms benefit from the parallelization?

On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
Mika's user avatar
  • 341
1 vote
1 answer
141 views

Are the two policies in SARSA for choosing an action the same?

Here is the pseudocode for SARSA (which I took from here) Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, ...
PeterBe's user avatar
  • 256
3 votes
0 answers
59 views

How to fix high variance of the returns on a 2d env?

I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution. It is basically a 2d game where you have to move a small circle around the screen and try to avoid ...
debrises's user avatar
2 votes
1 answer
630 views

How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?

I was trying to code the on-policy Monte Carlo control method. The initial policy chosen needs to be an $\epsilon$-soft policy. Can someone tell me how to code an $\epsilon$-soft policy? I know how to ...
A Q's user avatar
  • 23
3 votes
1 answer
178 views

Is it possible to apply a particular exploration policy for the on-policy RL agents?

Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy ...
Pulse9's user avatar
  • 282
6 votes
1 answer
306 views

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being \begin{equation} \mu(s) = \frac{\...
Felipe Costa's user avatar
1 vote
0 answers
1k views

PPO in continuous control not working

I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
Alexander Yukhimchuk's user avatar
3 votes
1 answer
143 views

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? Knowing that, my main question concerns Q-learning and off-policy methods: For me, there is something weird in ...
JeanMi's user avatar
  • 165
4 votes
1 answer
218 views

How should I generate datasets for a SARSA agent when the environment is not simple?

I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. ...
Shahin's user avatar
  • 153
5 votes
1 answer
253 views

Why does off-policy learning outperform on-policy learning?

I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works. I saw this in a book: Q-learning is an off-policy learner. An off-...
Exploring's user avatar
  • 343
4 votes
1 answer
156 views

What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that ...
Metrician's user avatar
1 vote
1 answer
323 views

Do we need multiple parallel environments to train in batches an on-policy algorithm?

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
Daniel's user avatar
  • 111
4 votes
1 answer
348 views

What is the difference between on-policy and off-policy for continuous environments?

I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space. First, some preliminary questions: in this case, what is ...
unter_983's user avatar
  • 331
1 vote
1 answer
547 views

Can we combine Off-Policy with On-Policy Algorithms?

On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms ...
Ray Walker's user avatar
2 votes
2 answers
2k views

Why is DDPG an off-policy RL algorithm?

In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
GoingMyWay's user avatar
1 vote
1 answer
909 views

What are the differences between 1-step SARSA and SARSA?

SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?
ycenycute's user avatar
  • 341
6 votes
1 answer
3k views

Is Expected SARSA an off-policy or on-policy algorithm?

I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one. Sutton and Barto's textbook describes Expected Sarsa thusly: In these cliff walking results Expected Sarsa was ...
Y. Xu's user avatar
  • 63
2 votes
0 answers
249 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
Luca Thiede's user avatar
2 votes
1 answer
152 views

Could we update the policy network with previous trajectories using supervised learning?

I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is ...
Ray Walker's user avatar
2 votes
0 answers
75 views

Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
Khush Agrawal's user avatar
6 votes
1 answer
786 views

Convergence of semi-gradient TD(0) with non-linear function approximation

I am looking for a result that shows the convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (...
srinivas tunuguntla's user avatar
14 votes
1 answer
8k views

What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

In the context of RL, there is the notion of on-policy and off-policy algorithms. I understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the notion of ...
nbro's user avatar
  • 40.8k
2 votes
1 answer
229 views

Why is the actor-critic algorithm limited to using on-policy data?

Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
apuffin's user avatar
  • 31
2 votes
1 answer
322 views

Do we need the transition probability function when calculating the importance sampling ratio?

I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused. There is one thing I ...
Manuel Pasieka's user avatar
4 votes
1 answer
3k views

Why is GLIE Monte-Carlo control an on-policy control?

In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control. But why is it an on-policy control? The sampling follows a policy $\pi$ while ...
fish_tree's user avatar
  • 247
6 votes
2 answers
2k views

What is the difference between on and off-policy deterministic actor-critic?

In the paper Deterministic Policy Gradient Algorithms, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic". I don't know what's the difference ...
fish_tree's user avatar
  • 247