Tag Info

Hot answers tagged actor-critic-methods

20 votes
Accepted

How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
• 4,007
18 votes
Accepted

What is the difference between actor-critic and advantage actor-critic?

Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of ...
• 9,419
14 votes

How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
• 3,597
8 votes
Accepted

Is reinforcement learning only about determining the value function?

There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn ...
• 470
7 votes

How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
• 143
6 votes

How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
• 495
6 votes
Accepted

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
• 24.4k
5 votes
Accepted

What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?

The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA: The ...
• 24.4k
5 votes
Accepted

What difference does it make whether Actor and Critic share the same network or not?

One can expect the optimal high-level features required to choose the next action and to evaluate a state to be quite similar. Because of that, it is a reasonable idea to share the same network for ...
• 111
5 votes

What are the advantages of RL with actor-critic methods over actor-only methods?

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods? One practical benefit is that critics can use TD learning to bootstrap, allowing them to ...
• 24.4k
4 votes
Accepted

What is the difference between on and off-policy deterministic actor-critic?

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different. The key to ...
• 522
4 votes
Accepted

Meaning of Actor Output in Actor Critic Reinforcement Learning

When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to ...
• 9,419
4 votes
Accepted

What is the purpose of the actor in actor-critic algorithms?

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used ...
• 24.4k
4 votes
Accepted

Where does entropy enter in Soft Actor-Critic?

In the answer I'll be using notation similar to the one from the SAC paper. If we look at the standard objective function for policy gradient methods we have \begin{align} J_\pi &= V_\pi(s_t)\\ &...
• 2,236
4 votes
Accepted

How can the derivative of a neural network be calculated, given no mathematical expression?

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate? Don't worry - lots of people get confused about this, because it seems ...
• 156
4 votes
Accepted

What is the intuition behind the TD(0) equation with average reward, and how is it derived?

This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t) \...
• 2,236
4 votes
Accepted

How does being on-policy prevent us from using the replay buffer with the policy gradients?

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is \begin{align} E_{\pi_b}[G_t|...
• 2,236
3 votes

How is parallelism implemented in RL algorithms like PPO?

OpenAI have a post on that: https://openai.com/blog/openai-five/ They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed ...
• 203
3 votes

How is the actor-critic algorithm guaranteed to converge?

There are different actor-critic (AC) algorithms with different convergence guarantees. For example, AC algorithms where the critic is tabular have different convergence guarantees than AC algorithms ...
• 34.3k
3 votes
Accepted

What is the gradient of the objective function in the Soft Actor-Critic paper?

I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me. I'll disregard expectation here to make things ...
• 2,236
3 votes

What is the difference between actor-critic and advantage actor-critic?

According to Sutton and Barto, they are the same thing. Note 13.5-6 (page 338) of their Reinforcement Learning: An Introduction, 2nd Edition book: Actor-critic methods are sometimes referred to as ...
3 votes

Would you categorize policy iteration as an actor-critic reinforcement learning approach?

Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not ...
• 24.4k
2 votes

How should I handle invalid actions (when using REINFORCE)?

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
• 143
2 votes

• 226
2 votes
Accepted

How to set the target for the actor in A2C?

In short, my last sentence was the correct answer. The "target" is a one-hot with the selected action, but there's a trick. A2C Loss Function A very crucial part of A2C implementation that I missed ...
• 161
2 votes

Advantage computed the wrong way?

Yeah, it seems like it's a wrong implementation. vals_ref_v is a matrix of 1 row, and 128 columns. value_v.detach() is a matrix of 128 row
• 33
2 votes
Accepted

Why isn't my implementation of A2C for the the atari pong game converging?

Here is the commit I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was ...
• 161

Only top scored, non community-wiki answers of a minimum length are eligible