17
votes
What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?
The concepts of on-policy vs off-policy and online vs offline are separate, but do interact to make certain combinations more feasible. When looking at this, it is worth also considering the ...
- 23.8k
6
votes
Accepted
What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?
This expression: $|\mathcal{A}(s)|$ means
$|\quad|$ the size of
$\mathcal{A}(s)$ the set of actions in state $s$
or more simply the number of actions allowed in the state.
This makes sense in the ...
- 23.8k
5
votes
Accepted
If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?
This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor.
The first is probably ...
- 876
5
votes
Accepted
Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?
I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?
The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not ...
- 23.8k
4
votes
Accepted
How should I generate datasets for a SARSA agent when the environment is not simple?
I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem
There is no difference in concept, which is why tic-tac-toe and maze problems are used ...
- 23.8k
4
votes
Why does off-policy learning outperform on-policy learning?
This post contains many answers that describe the difference between on-policy vs. off-policy.
Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-...
- 131
4
votes
Accepted
Why is GLIE Monte-Carlo control an on-policy control?
In this case, $\pi$ has always been an $\epsilon$-greedy policy. In every iteration, this $\pi$ is used to generate ($\epsilon$-greedily) a trajectory from which the new $Q(s, a)$ values are ...
- 522
4
votes
Accepted
What is the difference between on and off-policy deterministic actor-critic?
The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different.
The key to ...
- 522
3
votes
Accepted
Do we need the transition probability function when calculating the importance sampling ratio?
There is one thing I don't particularly understand. Why do we need the state-transition probability function when calculating the importance sampling ratio for off-policy prediction?
It is not needed ...
- 23.8k
3
votes
Why is DDPG an off-policy RL algorithm?
DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
- 4,055
3
votes
Accepted
Is Expected SARSA an off-policy or on-policy algorithm?
Expected SARSA can be used either on-policy or off-policy.
The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the ...
- 23.8k
2
votes
Accepted
Could we update the policy network with previous trajectories using supervised learning?
You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning)....
- 2,226
2
votes
Convergence of semi-gradient TD(0) with non-linear function approximation
Apparently there is an example of non-convergence for semi-gradient sarsa, according to Rich Sutton (check slide 35). I guess TD(0) is not so different. So, probably your approximator will need to ...
- 395
2
votes
Accepted
What is the difference between on-policy and off-policy for continuous environments?
First, some preliminary questions: in this case, what is the optimal policy?
It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of ...
- 23.8k
1
vote
Do we need multiple parallel environments to train in batches an on-policy algorithm?
We don't need multiple environments.
On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest ...
- 163
1
vote
Using reinforcement learning for human-robot interaction
Your question is difficult to answer because it is rather vague. Whether or not you can train an agent on your data depends on a few things. The two main things are: how much data do you have and how ...
- 205
1
vote
Are the two policies in SARSA for choosing an action the same?
For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode.
...
- 23.8k
1
vote
Accepted
How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?
You cannot code an $\epsilon$-soft policy directly, because it is not specific enough.
A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for ...
- 23.8k
1
vote
Accepted
Is it possible to apply a particular exploration policy for the on-policy RL agents?
In part it depends on the on-policy method you are using. In general you are not free to change the policy arbitrarily for on-policy policy gradient methods such as PPO or A3C.
However, if you are ...
- 23.8k
1
vote
Accepted
Can we combine Off-Policy with On-Policy Algorithms?
In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy ...
- 682
1
vote
Why is the actor-critic algorithm limited to using on-policy data?
It's because, in the actor-critic algorithm, the objective function is an expectation under the $\tau$ of the policy. If we want to use off-policy data, we have to resort to importance sampling ...
- 31
1
vote
What is the difference between on and off-policy deterministic actor-critic?
The main difference between on-policy and off-policy is how to get samples and what policy we optimize.
In off-policy deterministic actor-critic, the trajectories are samples from beta distribution (...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible