# Tag Info

Accepted

### How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
• 4,295
Accepted

### What is the difference between actor-critic and advantage actor-critic?

Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of ...
• 10.4k

### How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
• 3,837
Accepted

### Is reinforcement learning only about determining the value function?

There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn ...
• 590

### How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
• 163
Accepted

### What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?

The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA: The ...
• 32.7k

### How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
• 553
Accepted

### Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
• 32.7k
Accepted

### What difference does it make whether Actor and Critic share the same network or not?

One can expect the optimal high-level features required to choose the next action and to evaluate a state to be quite similar. Because of that, it is a reasonable idea to share the same network for ...
• 380
Accepted

### How can the derivative of a neural network be calculated, given no mathematical expression?

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate? Don't worry - lots of people get confused about this, because it seems ...
• 166
Accepted

### Meaning of Actor Output in Actor Critic Reinforcement Learning

When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to ...
• 10.4k
Accepted

### How does being on-policy prevent us from using the replay buffer with the policy gradients?

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is \begin{align} E_{\pi_b}[G_t|...
• 2,406

### What are the advantages of RL with actor-critic methods over actor-only methods?

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods? One practical benefit is that critics can use TD learning to bootstrap, allowing them to ...
• 32.7k
Accepted

### What is the difference between on and off-policy deterministic actor-critic?

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different. The key to ...
• 562
Accepted

### What is the gradient of the objective function in the Soft Actor-Critic paper?

I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me. I'll disregard expectation here to make things ...
• 2,406
Accepted

### What is the purpose of the actor in actor-critic algorithms?

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used ...
• 32.7k
Accepted

### Where does entropy enter in Soft Actor-Critic?

In the answer I'll be using notation similar to the one from the SAC paper. If we look at the standard objective function for policy gradient methods we have \begin{align} J_\pi &= V_\pi(s_t)\\ &...
• 2,406
Accepted

### What is the intuition behind the TD(0) equation with average reward, and how is it derived?

This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t) \...
• 2,406

### How is the actor-critic algorithm guaranteed to converge?

There are different actor-critic (AC) algorithms with different convergence guarantees. For example, AC algorithms where the critic is tabular have different convergence guarantees than AC algorithms ...
• 41k

### How is parallelism implemented in RL algorithms like PPO?

OpenAI have a post on that: https://openai.com/blog/openai-five/ They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed ...
• 203

### Would you categorize policy iteration as an actor-critic reinforcement learning approach?

Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not ...
• 32.7k

### What is the difference between actor-critic and advantage actor-critic?

According to Sutton and Barto, they are the same thing. Note 13.5-6 (page 338) of their Reinforcement Learning: An Introduction, 2nd Edition book: Actor-critic methods are sometimes referred to as ...

### How should I handle invalid actions (when using REINFORCE)?

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
• 165

• 226
Accepted

### How to set the target for the actor in A2C?

In short, my last sentence was the correct answer. The "target" is a one-hot with the selected action, but there's a trick. A2C Loss Function A very crucial part of A2C implementation that I missed ...
• 161