17 votes

What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

The concepts of on-policy vs off-policy and online vs offline are separate, but do interact to make certain combinations more feasible. When looking at this, it is worth also considering the ...
user avatar
  • 23.8k
6 votes
Accepted

What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

This expression: $|\mathcal{A}(s)|$ means $|\quad|$ the size of $\mathcal{A}(s)$ the set of actions in state $s$ or more simply the number of actions allowed in the state. This makes sense in the ...
user avatar
  • 23.8k
5 votes
Accepted

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor. The first is probably ...
user avatar
  • 876
5 votes
Accepted

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not ...
user avatar
  • 23.8k
4 votes
Accepted

How should I generate datasets for a SARSA agent when the environment is not simple?

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem There is no difference in concept, which is why tic-tac-toe and maze problems are used ...
user avatar
  • 23.8k
4 votes

Why does off-policy learning outperform on-policy learning?

This post contains many answers that describe the difference between on-policy vs. off-policy. Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-...
user avatar
  • 131
4 votes
Accepted

Why is GLIE Monte-Carlo control an on-policy control?

In this case, $\pi$ has always been an $\epsilon$-greedy policy. In every iteration, this $\pi$ is used to generate ($\epsilon$-greedily) a trajectory from which the new $Q(s, a)$ values are ...
user avatar
4 votes
Accepted

What is the difference between on and off-policy deterministic actor-critic?

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different. The key to ...
user avatar
3 votes
Accepted

Do we need the transition probability function when calculating the importance sampling ratio?

There is one thing I don't particularly understand. Why do we need the state-transition probability function when calculating the importance sampling ratio for off-policy prediction? It is not needed ...
user avatar
  • 23.8k
3 votes

Why is DDPG an off-policy RL algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
user avatar
3 votes
Accepted

Is Expected SARSA an off-policy or on-policy algorithm?

Expected SARSA can be used either on-policy or off-policy. The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the ...
user avatar
  • 23.8k
2 votes
Accepted

Could we update the policy network with previous trajectories using supervised learning?

You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning)....
user avatar
  • 2,226
2 votes

Convergence of semi-gradient TD(0) with non-linear function approximation

Apparently there is an example of non-convergence for semi-gradient sarsa, according to Rich Sutton (check slide 35). I guess TD(0) is not so different. So, probably your approximator will need to ...
user avatar
2 votes
Accepted

What is the difference between on-policy and off-policy for continuous environments?

First, some preliminary questions: in this case, what is the optimal policy? It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of ...
user avatar
  • 23.8k
1 vote

Do we need multiple parallel environments to train in batches an on-policy algorithm?

We don't need multiple environments. On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest ...
user avatar
  • 163
1 vote

Using reinforcement learning for human-robot interaction

Your question is difficult to answer because it is rather vague. Whether or not you can train an agent on your data depends on a few things. The two main things are: how much data do you have and how ...
user avatar
1 vote

Are the two policies in SARSA for choosing an action the same?

For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode. ...
user avatar
  • 23.8k
1 vote
Accepted

How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?

You cannot code an $\epsilon$-soft policy directly, because it is not specific enough. A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for ...
user avatar
  • 23.8k
1 vote
Accepted

Is it possible to apply a particular exploration policy for the on-policy RL agents?

In part it depends on the on-policy method you are using. In general you are not free to change the policy arbitrarily for on-policy policy gradient methods such as PPO or A3C. However, if you are ...
user avatar
  • 23.8k
1 vote
Accepted

Can we combine Off-Policy with On-Policy Algorithms?

In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy ...
user avatar
1 vote

Why is the actor-critic algorithm limited to using on-policy data?

It's because, in the actor-critic algorithm, the objective function is an expectation under the $\tau$ of the policy. If we want to use off-policy data, we have to resort to importance sampling ...
user avatar
  • 31
1 vote

What is the difference between on and off-policy deterministic actor-critic?

The main difference between on-policy and off-policy is how to get samples and what policy we optimize. In off-policy deterministic actor-critic, the trajectories are samples from beta distribution (...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible