Skip to main content
2 votes

It is not clear why SARSA is on-policy but Q-learning off-policy

Your mistake is your definition of on-policy. In both cases, we select actions based on our current action value estimates. It's how we select those actions that differentiates the 2 algorithms. An ...
nbro's user avatar
  • 40.9k
2 votes
Accepted

Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?

From the same page's pseudocode for off-policy Monte Carlo Control for estimating the optimal target policy, in the nested inner loop you have: If $A_t \neq \pi(S_t)$ then exit inner Loop (proceed to ...
cinch's user avatar
  • 2,277
2 votes

Why does HER not work with on-policy RL algorithms?

In principle replay mechanisms as HER, PER, etc cannot be applied to on-policy algorithms, like SARSA and Policy gradient as stated by @Neil Slater. Anyway, If you are willing to "adapt" ...
Luca Anzalone's user avatar
2 votes
Accepted

Why does HER not work with on-policy RL algorithms?

On policy algorithms contain policy and/or value update calculations that assume data was generated by the current policy. Breaking that assumption will cause them to miscalculate, or not function at ...
Neil Slater's user avatar
  • 32.7k
1 vote
Accepted

Does the off-policy/on-policy regime plays role in Contextual Bandits?

Say you have 2 deterministic slots, one that always returns 1 and the second that always returns 2 as rewards Your behavioural policy selects the first slot machines 80% of the times, thus the second ...
Alberto's user avatar
  • 2,293
1 vote
Accepted

A question about the reward calculation in the Hindsight Experience Replay algorithm

Meanwhile I can answer my own question. After figuring it out, what the HER algorithm really needs, I found out, that the right answer is 1: you need a reward function for calculating the new reward ...
Dave's user avatar
  • 182
1 vote
Accepted

How would one normalize observations in off-policy online reinforcement learning?

(I'll provide a couple of ideas, but I don't think these would fully address the problem.) If you normalize with running statistics (e.g. mean and std) as in the online case, you would get two ...
Luca Anzalone's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible