Questions tagged [advantage-actor-critic]

For questions related to the advantage actor-critic algorithms (that is, actor-critic algorithms that use the "advantage" function).

Filter by
Sorted by
Tagged with
0
votes
0answers
27 views

Why would the agent always take the same action in the test environment?

I'm trying to set up an agent with PPO2. But I've tried with: A2C DQN PPO2 However, in the test environment, the agent always takes the same action, and the total profit is negative. What can be the ...
0
votes
0answers
15 views

A2C value function outputs keep increasing

I was implementing the A2C algorithm with as close to baseline setup as possible, and this is the code I came up with. The problem is that even after multiple rechecks, the program isn't showing ...
0
votes
0answers
28 views

How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?

I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
-2
votes
1answer
90 views

Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions

i'm trying to solve a problem in which i need to carry out reinforcement learning with multiple simultaneous actions in continuous action space . i checked the multiagent structure; however, im trying ...
1
vote
0answers
31 views

What is the difference between step_model and train_model in the OpenAI implementation of the A2C algorithm?

I'm struggling a little with understanding the OpenAI implementation of A2C in the baselines (version 2.9.0) package. From my understanding, one ...
2
votes
0answers
50 views

Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: \begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} ...
3
votes
1answer
1k views

What is the difference between vanilla policy gradient and advantage actor-critic?

What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)? By vanilla policy gradient I am specifically referring to spinning up's ...
1
vote
1answer
64 views

Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...
4
votes
1answer
245 views

What is the advantage of using more than one environment with the advantage actor-critic?

make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4")) envs = [make_env() for _ in range(NUM_ENVS)] Here is a code you can look at. ...
0
votes
1answer
70 views

Is A2C loss function taking smaller steps for larger mistakes?

A2C loss is usually defined as advantage * (-log(actor_predictions)) * target where target is a one-hot vector (with some ...
3
votes
1answer
369 views

How to set the target for the actor in A2C?

I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, ...
0
votes
2answers
281 views

Why I got the same action when testing the A2C?

I'm working on an advantage actor-critic (A2C) reinforcement learning model, but when I test the model after I trained for 3500 episodes, I start to get almost the same action for all testing episodes....
3
votes
0answers
447 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
3
votes
1answer
349 views

What is the difference between A2C and running an agent in an environment vector?

I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector? I personally ...
16
votes
3answers
8k views

What is the difference between actor-critic and advantage actor-critic?

I'm struggling to understand the difference between actor-critic and advantage actor-critic. At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an ...