Questions tagged [advantage-actor-critic]

For questions related to the advantage actor-critic algorithms (that is, actor-critic algorithms that use the "advantage" function).

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Can A2C deal with a reward that is decided later than action selection?

I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ...
user77436's user avatar
0 votes
1 answer
86 views

Action space for the A2C algorithm

I am working on a problem where I need to generate a list of 0 - 8 different prices as the action space where generating 0 prices represents doing nothing ...
willem12's user avatar
0 votes
0 answers
56 views

A2C unable to solve Cartpole

I have coded my own A2C implementation using PyTorch. However, despite having followed the algorithm pseudo-code from several sources, my implementation is not able to achieve a proper Cartpole ...
Antonio's user avatar
  • 26
2 votes
1 answer
163 views

Why does Advantage Learning help function approximators?

Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps. For instance, the actor loss for a simple actor critic algorithm is given by - <...
desert_ranger's user avatar
2 votes
2 answers
2k views

What is the difference between A2C and Q-Learning, and when to use one over the other?

I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?
Hani's user avatar
  • 23
1 vote
1 answer
295 views

How to tune hypeparametes in A2C-ppo?

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m ...
voroshilov's user avatar
1 vote
0 answers
68 views

Can vanilla multi armed bandit problems be solved by RL algorithms like A2C and PPO?

Let's say we have N bandit machines with some distributions (assume some are gaussian, some are uniform, some are chi squared). We want to maximize rewards in X amount of time. I am aware that ...
Prasanjit Rath's user avatar
2 votes
2 answers
419 views

A2C: Why do episode rewards reset?

I am training a model using A2C with stable baselines 2. When I increased the timesteps I noticed that episode rewards seem to reset (see attached plot). I don´t understand where these sudden decays ...
qwertz's user avatar
  • 21
0 votes
0 answers
33 views

How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?

I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
Leon Jovanovic's user avatar
-1 votes
1 answer
578 views

Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions

i'm trying to solve a problem in which i need to carry out reinforcement learning with multiple simultaneous actions in continuous action space . i checked the multiagent structure; however, im trying ...
navid mohamadi's user avatar
1 vote
0 answers
39 views

What is the difference between step_model and train_model in the OpenAI implementation of the A2C algorithm?

I'm struggling a little with understanding the OpenAI implementation of A2C in the baselines (version 2.9.0) package. From my understanding, one ...
sjokkopudd's user avatar
7 votes
2 answers
3k views

What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?

What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)? By vanilla policy gradient I am specifically referring to spinning up's ...
Vedant Shah's user avatar
1 vote
1 answer
447 views

Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...
pranav's user avatar
  • 191
4 votes
1 answer
772 views

What is the advantage of using more than one environment with the advantage actor-critic?

make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4")) envs = [make_env() for _ in range(NUM_ENVS)] Here is a code you can look at. ...
jgauth's user avatar
  • 261
2 votes
0 answers
379 views

What is the original source of the TD Advantage Actor-Critic algorithm?

What is the original source of the TD Advantage Actor-Critic algorithm? I found this tutorial really helpful for learning the algorithm. However, what is the original source of this algorithm?
C.Neal's user avatar
  • 121
0 votes
1 answer
173 views

Is A2C loss function taking smaller steps for larger mistakes?

A2C loss is usually defined as advantage * (-log(actor_predictions)) * target where target is a one-hot vector (with some ...
asaf92's user avatar
  • 161
4 votes
1 answer
1k views

How to set the target for the actor in A2C?

I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, ...
asaf92's user avatar
  • 161
0 votes
2 answers
941 views

Why I got the same action when testing the A2C?

I'm working on an advantage actor-critic (A2C) reinforcement learning model, but when I test the model after I trained for 3500 episodes, I start to get almost the same action for all testing episodes....
I_Al-thamary's user avatar
3 votes
0 answers
1k views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my Reddit post: (Sorry if this does not fit here, please tell me and I delete it) Help regarding I'm working on an implementation of PPO, which I plan to use in my (Bachelors) Thesis. To ...
LJKS's user avatar
  • 131
4 votes
2 answers
557 views

What is the difference between A2C and running an agent in an environment vector?

I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector? I personally ...
Maybe's user avatar
  • 451
19 votes
4 answers
10k views

What is the difference between actor-critic and advantage actor-critic?

I'm struggling to understand the difference between actor-critic and advantage actor-critic. At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an ...
Blaszard's user avatar
  • 1,037