Questions tagged [advantage-actor-critic]
For questions related to the advantage actor-critic algorithms (that is, actor-critic algorithms that use the "advantage" function).
9
questions
1
vote
1answer
76 views
What is the difference between vanilla policy gradient and advantage actor-critic?
What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)?
By vanilla policy gradient I am specifically referring to spinning up's ...
0
votes
0answers
24 views
How to deal with GAE ineffectiveness because of critic value adaptation?
I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a ...
1
vote
1answer
42 views
Why is the “reward to go” replaced by Q instead of V, when transitioning from PG to actor critic methods?
While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5).
I am not ...
11
votes
3answers
6k views
What is the difference between actor-critic and advantage actor-critic?
I'm struggling to understand the difference between actor-critic and advantage actor-critic.
At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an ...
3
votes
1answer
117 views
What is the advantage of using more than one environment with the advantage actor-critic?
make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4"))
envs = [make_env() for _ in range(NUM_ENVS)]
Here is a code you can look at.
...
2
votes
1answer
216 views
What is the difference between A2C and running an agent in an environment vector?
I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector?
I personally ...
0
votes
1answer
32 views
Is A2C loss function taking smaller steps for larger mistakes?
A2C loss is usually defined as advantage * (-log(actor_predictions)) * target where target is a one-hot vector (with some ...
2
votes
1answer
114 views
How to set the target for the actor in A2C?
I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, ...
2
votes
0answers
307 views
Implementation of PPO - Value Loss not converging, return plateauing
Copy from my reddit post:
(Sorry if this does not fit here, please tell me and i delete it)
Help regarding
I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
