21
votes
Accepted
How should I handle invalid actions (when using REINFORCE)?
Just ignore the invalid moves.
For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
- 4,205
18
votes
Accepted
What is the difference between actor-critic and advantage actor-critic?
Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of ...
- 10.1k
14
votes
How should I handle invalid actions (when using REINFORCE)?
Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
- 3,787
8
votes
Accepted
Is reinforcement learning only about determining the value function?
There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn ...
- 570
7
votes
How should I handle invalid actions (when using REINFORCE)?
I faced a similar issue recently with Minesweeper.
The way I solved it was by ignoring the illegal/invalid moves entirely.
Use the Q-network to predict the Q-values for all of your actions (valid ...
- 163
7
votes
Accepted
What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?
The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:
The ...
- 30.3k
6
votes
How should I handle invalid actions (when using REINFORCE)?
IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
- 553
6
votes
Accepted
Can I apply DQN or policy gradient algorithms in the contextual bandit setting?
MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance.
Therefore, methods used in RL to solve ...
- 30.3k
6
votes
Accepted
What difference does it make whether Actor and Critic share the same network or not?
One can expect the optimal high-level features required to choose the next action and to evaluate a state to be quite similar. Because of that, it is a reasonable idea to share the same network for ...
- 380
5
votes
Accepted
How can the derivative of a neural network be calculated, given no mathematical expression?
I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate?
Don't worry - lots of people get confused about this, because it seems ...
- 166
5
votes
Accepted
Meaning of Actor Output in Actor Critic Reinforcement Learning
When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to ...
- 10.1k
5
votes
Accepted
How does being on-policy prevent us from using the replay buffer with the policy gradients?
Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is
\begin{align}
E_{\pi_b}[G_t|...
- 2,336
5
votes
What are the advantages of RL with actor-critic methods over actor-only methods?
In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?
One practical benefit is that critics can use TD learning to bootstrap, allowing them to ...
- 30.3k
4
votes
Accepted
What is the difference between on and off-policy deterministic actor-critic?
The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different.
The key to ...
- 552
4
votes
Accepted
What is the purpose of the actor in actor-critic algorithms?
For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms?
In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used ...
- 30.3k
4
votes
Accepted
Where does entropy enter in Soft Actor-Critic?
In the answer I'll be using notation similar to the one from the SAC paper.
If we look at the standard objective function for policy gradient methods we have
\begin{align}
J_\pi &= V_\pi(s_t)\\
&...
- 2,336
4
votes
Accepted
What is the intuition behind the TD(0) equation with average reward, and how is it derived?
This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as
\begin{equation}
TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t)
\...
- 2,336
3
votes
How is the actor-critic algorithm guaranteed to converge?
There are different actor-critic (AC) algorithms with different convergence guarantees. For example, AC algorithms where the critic is tabular have different convergence guarantees than AC algorithms ...
- 39.6k
3
votes
How is parallelism implemented in RL algorithms like PPO?
OpenAI have a post on that: https://openai.com/blog/openai-five/
They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed ...
- 203
3
votes
Accepted
What is the gradient of the objective function in the Soft Actor-Critic paper?
I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me.
I'll disregard expectation here to make things ...
- 2,336
3
votes
Would you categorize policy iteration as an actor-critic reinforcement learning approach?
Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not ...
- 30.3k
2
votes
What is the difference between actor-critic and advantage actor-critic?
According to Sutton and Barto, they are the same thing. Note 13.5-6 (page 338) of their Reinforcement Learning: An Introduction, 2nd Edition book:
Actor-critic methods are sometimes referred to as ...
2
votes
How should I handle invalid actions (when using REINFORCE)?
An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
- 165
2
votes
How do I calculate the policy in the Proximal Policy Optimization algorithm?
You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\...
- 266
2
votes
What is the difference between A2C and running an agent in an environment vector?
I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different ...
- 2,336
2
votes
A2C Critic Loss Interpretation
Unfortunately no, the way to go is track the total reward and see if it's increasing and converging eventually. Value loss isn't a useful metric as the loss can be 0 when the value network always ...
- 173
2
votes
Accepted
Why not more TD(𝜆) in actor-critic algorithms?
Theoretically, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially ...
- 226
2
votes
Accepted
How to set the target for the actor in A2C?
In short, my last sentence was the correct answer. The "target" is a one-hot with the selected action, but there's a trick.
A2C Loss Function
A very crucial part of A2C implementation that I missed ...
- 161
2
votes
Accepted
What is the advantage of using more than one environment with the advantage actor-critic?
What is the advantage of using more than one environment with a single agent?
There are two main advantages to this approach:
The dataset for training is closer to the independent, identically ...
- 30.3k
Only top scored, non community-wiki answers of a minimum length are eligible