Questions tagged [on-policy-methods]

For questions related to the "on-policy" reinforcement learning algorithms.

Filter by
Sorted by
Tagged with
6
votes
1answer
81 views

Understanding the On-policy state distribution for episodic tasks with $\gamma \in (0,1)$

In Sutton and Barto's Reinforcement Learning: An Introduction, section 9.2 (page 199) (here is a screenshot) describes the on-policy distribution in episodic tasks, with $\gamma =1$, as being \begin{...
1
vote
0answers
85 views

PPO in continuous control not working

I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
3
votes
1answer
75 views

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? Knowing that, my main question concerns Q-learning and off-policy methods: For me, there is something weird in ...
4
votes
1answer
132 views

How should I generate datasets for a SARSA agent when the environment is not simple?

I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. ...
3
votes
1answer
93 views

Why does off-policy learning outperform on-policy learning?

I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works. I saw this in a book: Q-learning is an off-policy learner. An off-...
4
votes
1answer
92 views

What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that ...
1
vote
1answer
77 views

Do we need multiple parallel environments to train in batches an on-policy algorithm?

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
3
votes
1answer
104 views

What is the difference between on-policy and off-policy for continuous environments?

I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space. First, some preliminary questions: in this case, what is ...
1
vote
1answer
114 views

Can we combine Off-Policy with On-Policy Algorithms?

On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms ...
2
votes
2answers
368 views

Why is DDPG an off-policy RL algorithm?

In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
1
vote
1answer
237 views

What are the differences between 1-step SARSA and SARSA?

SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?
4
votes
1answer
846 views

Is Expected SARSA an off-policy or on-policy algorithm?

I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one. Sutton and Barto's textbook describes Expected Sarsa thusly: In these cliff walking results Expected Sarsa was ...
2
votes
0answers
73 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
2
votes
1answer
54 views

Could we update the policy network with previous trajectories using supervised learning?

I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is ...
2
votes
0answers
42 views

Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
1
vote
0answers
31 views

How would you differentiate between different on-policy reinforcement learning algorithms?

How would you differentiate between different on-policy reinforcement learning algorithms?
4
votes
1answer
198 views

Convergence of semi-gradient TD(0) with non-linear function approximation

I am looking for a result that shows the convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (...
6
votes
1answer
3k views

What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

In the context of RL, there is the notion of on-policy and off-policy algorithms. I understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the notion of ...
2
votes
1answer
157 views

Why is the actor-critic algorithm limited to using on-policy data?

Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
2
votes
1answer
129 views

Do we need the transition probability function when calculating the importance sampling ratio?

I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused. There is one thing I ...
4
votes
1answer
2k views

Why is GLIE Monte-Carlo control an on-policy control?

In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control. But why is it an on-policy control? The sampling follows a policy $\pi$ while ...
6
votes
2answers
1k views

What is the difference between on and off-policy deterministic actor-critic?

In the paper Deterministic Policy Gradient Algorithms, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic". I don't know what's the difference ...