Questions tagged [actor-critic-methods]

For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Is there a multi-task RL algorithm that supports different action spaces for each agent?

I'm currently working on a project in which I need apply multi-task reinforcement learning. Over the same state space, each agent aims to do a separate task, but the action spaces of agents are ...
user avatar
  • 1
1 vote
0 answers
47 views

Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
user avatar
  • 49
0 votes
1 answer
60 views

Does Value Loss in Actor Critic not decrease at all?

I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://...
user avatar
1 vote
0 answers
51 views

How to use Actor-Critic RL with a categorical, state-dependent action space?

I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
user avatar
  • 266
4 votes
0 answers
679 views

Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow: $$ V_\pi(s_t)...
user avatar
1 vote
1 answer
72 views

How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?

Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
user avatar
  • 121
0 votes
0 answers
21 views

multi agent deep deterministic policy gradient for discrete actions

I am solving a multi agent problem where each agent has a critic and actor. The problem I am solving has discrete actions and discrete states. I came cross multi-agent deep deterministic policy ...
user avatar
1 vote
1 answer
60 views

Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
user avatar
  • 143
1 vote
0 answers
34 views

Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
user avatar
  • 11
1 vote
1 answer
224 views

PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
user avatar
  • 203
0 votes
0 answers
29 views

How to train critic network in PPO with multiple actor?

I try to code an algorithm to control a robot's movement with continuous action space. I use this question and create an actor network for each action.How can policy gradients be applied in the case ...
user avatar
  • 3
0 votes
0 answers
53 views

tanh activation function output is not between -1 and 1 for continuous action PPO

I am using RLlib's (Ray = 1.4.0) PPO policy, and my first layer after the input (Conv layer) is producing a strange output keeping in mind that the activation for the output is a tanh, which I do ...
user avatar
  • 203
0 votes
0 answers
215 views

RLlib's Multi-agent PPO continuous actions turn into nan

After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
user avatar
  • 203
0 votes
0 answers
43 views

Actor-critic reinforcement learning updates and episode length

I am currently using a TD3 agent-critic network to control a vehicle suspension system, where the reward (or rather a penalty) is based on the vertical acceleration of the mass and is calculated at ...
user avatar
2 votes
0 answers
273 views

How to interpret the training loss curves in Soft-Actor-Critic (SAC)?

I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
user avatar
  • 35
0 votes
0 answers
24 views

How to obtain STD from Neural Network with 2 continuous action output

In my Environment, I have two continuous action space self.action_space = spaces.Box(low=np.array([0.,0.]), high=np.array([4.,0.02]), shape=(2,), dtype=np.float32) ...
user avatar
0 votes
0 answers
29 views

How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?

I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
user avatar
3 votes
0 answers
149 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
user avatar
1 vote
0 answers
23 views

Understanding advantage estimator in proximal policy optimization

I was reading Proximal Policy Optimization paper. It states following: The advantage estimator used is: $\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
user avatar
  • 211
0 votes
1 answer
161 views

Can I train a DQN on the same dataset for multiple epochs?

I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a ...
user avatar
0 votes
1 answer
79 views

Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)

I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action. $Q(s,...
user avatar
3 votes
0 answers
77 views

How to implement REINFORCE with eligibility traces?

The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
user avatar
6 votes
1 answer
371 views

What are the advantages of RL with actor-critic methods over actor-only methods?

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods? This is not a comparison with the Q-learning series, but probably a method of learning ...
user avatar
1 vote
0 answers
30 views

Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?

I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard. My ...
user avatar
  • 26
4 votes
1 answer
584 views

What difference does it make whether Actor and Critic share the same network or not?

I'm learning about Actor-Critic reinforcement learning algorithms. One source I encountered mentioned that Actor and Critic can either share one network (but use different output layers) or they can ...
user avatar
  • 663
1 vote
1 answer
126 views

Advantage Actor Critic model implementation with Tensorflowjs

I am trying to implement an Actor Critic method that controls an RC car. For this I have implemented a simulated environment and actor critic tensorflowjs models. My intention is to train a model to ...
user avatar
5 votes
1 answer
189 views

Is reinforcement learning only about determining the value function?

I started reading some reinforcement learning literature, and it seems to me that all approaches to solving reinforcement learning problems are about finding the value function (state-value function ...
user avatar
  • 287
2 votes
0 answers
59 views

Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: \begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} ...
user avatar
  • 337
0 votes
0 answers
61 views

Variance of the Gaussian policy is not decreasing while training the agent using Soft Actor-Critic method

I've written my own version of SAC(v2) for a problem with continuous action space. While training, the losses for the value network and both q functions steadily decrease down to 0.02-0.03. The loss ...
user avatar
1 vote
0 answers
73 views

How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?

For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer ...
user avatar
  • 111
0 votes
1 answer
75 views

Why would the reward of A3C with LSTM suddenly drop off after many episodes?

I am training an A3C with stacked LSTM. During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
user avatar
0 votes
1 answer
311 views

How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?

I'm working in A2C and I have an environment where there is increasing or decreasing in the number of agents. The action space in the environment will not change but the state will change when new ...
user avatar
0 votes
0 answers
65 views

How to modify the Actor-Critic policy gradient algorithm to perform Safe exploration in Reinforcement Learning

I am trying to implement safe exploration technique in [Ref.1]. I am using Soft Actor-Critic algorithm to teach an agent to introduce a bias between 0 and 1 to a specific state of interest in my ...
user avatar
0 votes
1 answer
29 views

Learning only using off-policy samples

When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a ...
user avatar
  • 101
1 vote
0 answers
41 views

When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?

I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this. Now, I want to know that for a problem/game like Breakout, where ...
user avatar
4 votes
1 answer
1k views

What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?

What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)? By vanilla policy gradient I am specifically referring to spinning up's ...
user avatar
2 votes
0 answers
54 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
user avatar
  • 121
1 vote
0 answers
55 views

Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case? I am a bit confused about the off-policy case (right figure). ...
user avatar
0 votes
1 answer
114 views

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be. Is it a scalar that I have to use to update all the weights (so all ...
user avatar
  • 281
1 vote
1 answer
86 views

Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...
user avatar
  • 181
0 votes
1 answer
518 views

Why is my Soft Actor-Critic's policy and value function losses not converging?

I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging,...
user avatar
  • 3
3 votes
1 answer
700 views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
user avatar
  • 155
1 vote
0 answers
105 views

Actor-Critic implementation not learning

I've implemented a vanilla actor-critic and have run into a wall. My model does not seem to be learning the optimal policy. The red graph below shows its performance in cartpole, where the algorithm ...
user avatar
2 votes
1 answer
177 views

Learning policy where action involves discrete and continuous parameters

Typically it seems like reinforcement learning involves learning over either a discrete or a continuous action space. An example might be choosing from a set of pre-defined game actions in Gym Retro ...
user avatar
  • 121
2 votes
1 answer
808 views

Why isn't my implementation of A2C for the the atari pong game converging?

I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different. https://colab.research.google.com/drive/...
user avatar
  • 161
2 votes
2 answers
195 views

Advantage computed the wrong way?

Here is the code written by Maxim Lapan. I am reading his book (Deep Reinforcement Learning Hands-on). I have seen a line in his code which is really weird. In the accumulation of the policy gradient $...
user avatar
  • 161
4 votes
1 answer
409 views

Would you categorize policy iteration as an actor-critic reinforcement learning approach?

One way of understanding the difference between value function approaches, policy approaches and actor-critic approaches in reinforcement learning is the following: A critic explicitly models a value ...
user avatar
  • 81
5 votes
1 answer
390 views

How does being on-policy prevent us from using the replay buffer with the policy gradients?

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we discussed in ...
user avatar
  • 161
2 votes
1 answer
56 views

Why a single trajectory can be used to update the policy network $\theta$ in A3C?

The Deep RL bootcamp on policy gradient techniques gives the update equation for the policy network in A3C as $\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}...
user avatar
  • 1,201
1 vote
1 answer
153 views

Once the environments are vectorized, how do I have to gather immediate experiences for the agent?

My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the ...
user avatar
  • 161