Questions tagged [actor-critic-methods]

For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).

Filter by
Sorted by
Tagged with
4 votes
1 answer
134 views

Notation used in paper on Continuous Time Reinforcement Learning

I am working on implementing the learning shown in this paper: https://homes.cs.washington.edu/~todorov/courses/amath579/reading/Continuous.pdf In the paper, the authors devise a continuous time ...
Derick Diana's user avatar
0 votes
0 answers
30 views

Non differentiable loss function train with actor critic style

I'm working on a project where a non differentiable loss is there. I'm thinking about how should I deal with them. My model is a very big lstm model (about 1M parameter), and after 500 steps (not sure ...
TWTom's user avatar
  • 13
0 votes
1 answer
307 views

Is PPO a policy-based method or an actor-critique-based method?

as far as i understand there are 3 categories of Reinforcement algorithms: Value-based methods (like DQN or Sarsa) Policy-based methods (like REINFORCE) Actor-critic-based methods (like A2C) To ...
PeterBe's user avatar
  • 256
0 votes
0 answers
61 views

How and whether to apply Reinforcement Learning in an Environment with a precise and always available Evaluation?

Say we want to train an agent $A$ in an environment $E$ which provides a continuous loss $L$. That is, we want $A$ to choose its actions $a$ so that it minimizes the mistake it does, i.e., it ...
Mathy's user avatar
  • 143
0 votes
2 answers
68 views

In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?

Taking as an example the Advantage Actor Critic, the objective function is: \begin{equation} \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\...
user avatar
0 votes
1 answer
46 views

Why is my agent stuck on the same action in my Twin Delayed Deep Deterministic Policy Gradient (TD3) program?

I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to ...
CloudZero's user avatar
0 votes
0 answers
150 views

Unable to interpret DDPG actor-critic loss curves

I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
davipeix's user avatar
0 votes
0 answers
29 views

actor critic gradient inconsistent with mathematical proofs

From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to $\theta$ with baseline is proved as: $$\bigtriangledown_\theta \eta(\...
Hooman's user avatar
  • 1
1 vote
1 answer
594 views

Reinforcement Learning - Why is actor loss in Actor-Critic multiplied by -1?

Reading through the TensorFlow guide for Actor-Critic learning, I saw that the actor loss is multiplied by -1 when calculating: The guide says this is to maximize the probabilities of actions with ...
jasooney23's user avatar
1 vote
1 answer
213 views

Getting always the same action on an A2C from stable_baselines3

I'm quite new to RL and have been trying to train an A2C model from stable_baselines3 to derive an integer sequence based on 3 other input sequences of floats. I have a custom gym environment that ...
Jesuspc's user avatar
  • 151
2 votes
1 answer
118 views

Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?

I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
XiaoBanni's user avatar
0 votes
0 answers
48 views

How are Target Values Generated in Alpha Zero Architecture

I am a little confused as to how the target values are generated to train the neural network with the Alpha Zero architecture(in specific to a chess game). I understand how the improved policy is ...
Kiran Manicka's user avatar
1 vote
0 answers
44 views

Role of $f$ Target Network in DDPG

I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
Vera Leighton 's user avatar
0 votes
1 answer
68 views

Should Cart Pole be immune to Reward Shifting?

I’ve been toying around with cart pole swingup, and have been flummoxed by a problem related to reward shifts. Traditionally, cart pole gives you positive reward if you’re stable and zero reward if ...
Mageek's user avatar
  • 111
2 votes
1 answer
332 views

Why is soft actor critic an off policy scheme?

I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy ...
ZZ1's user avatar
  • 35
1 vote
1 answer
382 views

Why is Soft Q Learning not an Actor Critic method?

I've been reading these two papers from Haarnoja et. al.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Reinforcement Learning with Deep Energy-...
frances_farmer's user avatar
1 vote
1 answer
58 views

Can action be dominated by state features in actor-critic algorithms?

I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I ...
Mika's user avatar
  • 341
0 votes
0 answers
179 views

Why do Soft Actor-Critic with automatic temperature tuning use only a single dual variable?

In section 5 of the paper “Soft Actor Critic Algorithms and Applications”, the authors propose to optimize the policy subject to the constraints that the entropy of action distribution should be ...
Cloudy's user avatar
  • 223
1 vote
0 answers
138 views

Has anyone here tried to implement MADDPG for a different environment and succeeded?

Has anyone tried implementing the multi-agent RL algorithm MADDPG (I've linked the paper below)? The paper seems to have a good amount citations, and they do have their code on github. However, a few ...
Confuse's user avatar
  • 111
3 votes
1 answer
190 views

Why does A2C use the actual returns from an episode in calculating the advantage?

Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value? Basically, why this: $A(s,a) = \sum_t\gamma^tr_t - V(s)...
JacKeown's user avatar
  • 125
2 votes
1 answer
859 views

Pytorch's Actor-critic implementation seems to be implemented in a Monte-Carlo fashion - why?

In the Actor-Critic example, provided by PyTorch, it seems that the update rule only occurs when the episode ends (like in a Monte-Carlo process). Specifically, in their ...
Hadar Sharvit's user avatar
1 vote
1 answer
368 views

How the Critic is used to train the Actor in Actor-Critic network

I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward. But I don't fully understand how the critic output (...
nrofis's user avatar
  • 111
2 votes
1 answer
285 views

What makes TRPO an actor-critic method? Where is the critic?

From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
thesofakillers's user avatar
0 votes
1 answer
69 views

Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?

My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
Magi Feeney's user avatar
1 vote
0 answers
56 views

Linear Actor Critic for continuing task and 1 continuous action => Any comment?

I wish to implement an Actor Critic agent using linear functions for a continuing task with one continuous action. Below the resulting pseudo-code I have reached by my own (the initialization part is ...
brz's user avatar
  • 11
0 votes
0 answers
2k views

Tensorflow-gpu and multiprocessing

I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds ...
Lyn Cassidy's user avatar
0 votes
0 answers
42 views

As someone starting out in RL, could you help me understand the differences between actor-only, critic-only, and actor-critic methods?

I have been reading some medium articles and these three methods pop up a lot. I am wondering what the differences between these are, what are the advantages of one over the other, etc. Also from my ...
No-Time-To-Day's user avatar
4 votes
2 answers
359 views

How do we estimate the value of a stochastic policy?

I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
mac_or_cheese's user avatar
1 vote
1 answer
529 views

How to implement PPO without using a Critic

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
Aeryan's user avatar
  • 53
1 vote
0 answers
116 views

DDPG agent with convolutional layers for feature extraction [closed]

I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I ...
Andreas Karatzas's user avatar
0 votes
1 answer
149 views

Replay buffer action range in DDPG

I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-...
goldfinch's user avatar
1 vote
0 answers
251 views

Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
Kronic's user avatar
  • 49
1 vote
1 answer
930 views

Does Value Loss in Actor Critic not decrease at all?

I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://...
Khabbab Zakaria's user avatar
1 vote
0 answers
161 views

How to use Actor-Critic RL with a categorical, state-dependent action space?

I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
profPlum's user avatar
  • 424
5 votes
1 answer
1k views

Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow: $$ V_\pi(s_t)...
Green Falcon's user avatar
1 vote
1 answer
503 views

How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?

Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
cerebrou's user avatar
  • 151
2 votes
1 answer
294 views

Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
Sanyou's user avatar
  • 165
1 vote
0 answers
61 views

Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
jazz's user avatar
  • 11
2 votes
2 answers
2k views

PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
hridayns's user avatar
  • 233
0 votes
0 answers
587 views

RLlib's Multi-agent PPO continuous actions turn into nan

After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
hridayns's user avatar
  • 233
3 votes
0 answers
2k views

How to interpret the training loss curves in Soft-Actor-Critic (SAC)?

I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
Manuel's user avatar
  • 45
0 votes
0 answers
34 views

How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?

I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
Leon Jovanovic's user avatar
3 votes
0 answers
411 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
user1779362's user avatar
1 vote
0 answers
97 views

Understanding advantage estimator in proximal policy optimization

I was reading Proximal Policy Optimization paper. It states following: The advantage estimator used is: $\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
Rnj's user avatar
  • 221
0 votes
1 answer
352 views

Can I train a DQN on the same dataset for multiple epochs?

I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a ...
Kyle Dixon's user avatar
0 votes
1 answer
466 views

Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)

I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action. $Q(s,...
Anubhav Sachan's user avatar
3 votes
0 answers
256 views

How to implement REINFORCE with eligibility traces?

The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
Javier Ventajas Hernández's user avatar
6 votes
1 answer
1k views

What are the advantages of RL with actor-critic methods over actor-only methods?

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods? This is not a comparison with the Q-learning series, but probably a method of learning ...
ground clown's user avatar
1 vote
0 answers
74 views

Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?

I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard. My ...
Kristof's user avatar
  • 26
6 votes
1 answer
2k views

What difference does it make whether Actor and Critic share the same network or not?

I'm learning about Actor-Critic reinforcement learning algorithms. One source I encountered mentioned that Actor and Critic can either share one network (but use different output layers) or they can ...
mark mark's user avatar
  • 763