Questions tagged [actor-critic-methods]
For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).
122
questions
4
votes
1
answer
134
views
Notation used in paper on Continuous Time Reinforcement Learning
I am working on implementing the learning shown in this paper:
https://homes.cs.washington.edu/~todorov/courses/amath579/reading/Continuous.pdf
In the paper, the authors devise a continuous time ...
0
votes
0
answers
30
views
Non differentiable loss function train with actor critic style
I'm working on a project where a non differentiable loss is there. I'm thinking about how should I deal with them.
My model is a very big lstm model (about 1M parameter), and after 500 steps (not sure ...
0
votes
1
answer
307
views
Is PPO a policy-based method or an actor-critique-based method?
as far as i understand there are 3 categories of Reinforcement algorithms:
Value-based methods (like DQN or Sarsa)
Policy-based methods (like REINFORCE)
Actor-critic-based methods (like A2C)
To ...
0
votes
0
answers
61
views
How and whether to apply Reinforcement Learning in an Environment with a precise and always available Evaluation?
Say we want to train an agent $A$ in an environment $E$ which provides a continuous loss $L$. That is, we want $A$ to choose its actions $a$ so that it minimizes the mistake it does, i.e., it ...
0
votes
2
answers
68
views
In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?
Taking as an example the Advantage Actor Critic, the objective function is:
\begin{equation}
\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\...
0
votes
1
answer
46
views
Why is my agent stuck on the same action in my Twin Delayed Deep Deterministic Policy Gradient (TD3) program?
I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to ...
0
votes
0
answers
150
views
Unable to interpret DDPG actor-critic loss curves
I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
0
votes
0
answers
29
views
actor critic gradient inconsistent with mathematical proofs
From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to $\theta$ with baseline is proved as:
$$\bigtriangledown_\theta \eta(\...
1
vote
1
answer
594
views
Reinforcement Learning - Why is actor loss in Actor-Critic multiplied by -1?
Reading through the TensorFlow guide for Actor-Critic learning, I saw that the actor loss is multiplied by -1 when calculating:
The guide says this is to maximize the probabilities of actions with ...
1
vote
1
answer
213
views
Getting always the same action on an A2C from stable_baselines3
I'm quite new to RL and have been trying to train an A2C model from stable_baselines3 to derive an integer sequence based on 3 other input sequences of floats. I have a custom gym environment that ...
2
votes
1
answer
118
views
Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
0
votes
0
answers
48
views
How are Target Values Generated in Alpha Zero Architecture
I am a little confused as to how the target values are generated to train the neural network with the Alpha Zero architecture(in specific to a chess game). I understand how the improved policy is ...
1
vote
0
answers
44
views
Role of $f$ Target Network in DDPG
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
0
votes
1
answer
68
views
Should Cart Pole be immune to Reward Shifting?
I’ve been toying around with cart pole swingup, and have been flummoxed by a problem related to reward shifts. Traditionally, cart pole gives you positive reward if you’re stable and zero reward if ...
2
votes
1
answer
332
views
Why is soft actor critic an off policy scheme?
I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy ...
1
vote
1
answer
382
views
Why is Soft Q Learning not an Actor Critic method?
I've been reading these two papers from Haarnoja et. al.:
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
Learning with a Stochastic Actor
Reinforcement Learning with Deep Energy-...
1
vote
1
answer
58
views
Can action be dominated by state features in actor-critic algorithms?
I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I ...
0
votes
0
answers
179
views
Why do Soft Actor-Critic with automatic temperature tuning use only a single dual variable?
In section 5 of the paper “Soft Actor Critic Algorithms and Applications”, the authors propose to optimize the policy subject to the constraints that the entropy of action distribution should be ...
1
vote
0
answers
138
views
Has anyone here tried to implement MADDPG for a different environment and succeeded?
Has anyone tried implementing the multi-agent RL algorithm MADDPG (I've linked the paper below)? The paper seems to have a good amount citations, and they do have their code on github. However, a few ...
3
votes
1
answer
190
views
Why does A2C use the actual returns from an episode in calculating the advantage?
Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?
Basically, why this:
$A(s,a) = \sum_t\gamma^tr_t - V(s)...
2
votes
1
answer
859
views
Pytorch's Actor-critic implementation seems to be implemented in a Monte-Carlo fashion - why?
In the Actor-Critic example, provided by PyTorch, it seems that the update rule only occurs when the episode ends (like in a Monte-Carlo process). Specifically, in their ...
1
vote
1
answer
368
views
How the Critic is used to train the Actor in Actor-Critic network
I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.
But I don't fully understand how the critic output (...
2
votes
1
answer
285
views
What makes TRPO an actor-critic method? Where is the critic?
From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
0
votes
1
answer
69
views
Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?
My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
1
vote
0
answers
56
views
Linear Actor Critic for continuing task and 1 continuous action => Any comment?
I wish to implement an Actor Critic agent using linear functions for a continuing task with one continuous action. Below the resulting pseudo-code I have reached by my own (the initialization part is ...
0
votes
0
answers
2k
views
Tensorflow-gpu and multiprocessing
I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds ...
0
votes
0
answers
42
views
As someone starting out in RL, could you help me understand the differences between actor-only, critic-only, and actor-critic methods?
I have been reading some medium articles and these three methods pop up a lot. I am wondering what the differences between these are, what are the advantages of one over the other, etc. Also from my ...
4
votes
2
answers
359
views
How do we estimate the value of a stochastic policy?
I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
1
vote
1
answer
529
views
How to implement PPO without using a Critic
I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
1
vote
0
answers
116
views
DDPG agent with convolutional layers for feature extraction [closed]
I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I ...
0
votes
1
answer
149
views
Replay buffer action range in DDPG
I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-...
1
vote
0
answers
251
views
Why is an action-independent baseline required to reduce variance?
I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
1
vote
1
answer
930
views
Does Value Loss in Actor Critic not decrease at all?
I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://...
1
vote
0
answers
161
views
How to use Actor-Critic RL with a categorical, state-dependent action space?
I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
5
votes
1
answer
1k
views
Why isn't a target network used for the critic in on-policy actor-critic methods?
Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow:
$$
V_\pi(s_t)...
1
vote
1
answer
503
views
How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
2
votes
1
answer
294
views
Joined vs Separate optimizer for Actor-Critic
Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
1
vote
0
answers
61
views
Setting initial values in DDPG to favor better actions
I'm working on a problem using DDPG.
Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
2
votes
2
answers
2k
views
PPO when does the update happen?
In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
0
votes
0
answers
587
views
RLlib's Multi-agent PPO continuous actions turn into nan
After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
3
votes
0
answers
2k
views
How to interpret the training loss curves in Soft-Actor-Critic (SAC)?
I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
0
votes
0
answers
34
views
How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?
I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
3
votes
0
answers
411
views
How to deal with a moving target in the Lunar Lander environment with DDPG?
I have noticed that DDPG does rather well at solving environments with a static target.
For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
1
vote
0
answers
97
views
Understanding advantage estimator in proximal policy optimization
I was reading Proximal Policy Optimization paper. It states following:
The advantage estimator used is:
$\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
0
votes
1
answer
352
views
Can I train a DQN on the same dataset for multiple epochs?
I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a ...
0
votes
1
answer
466
views
Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)
I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action.
$Q(s,...
3
votes
0
answers
256
views
How to implement REINFORCE with eligibility traces?
The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
6
votes
1
answer
1k
views
What are the advantages of RL with actor-critic methods over actor-only methods?
In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?
This is not a comparison with the Q-learning series, but probably a method of learning ...
1
vote
0
answers
74
views
Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?
I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard.
My ...
6
votes
1
answer
2k
views
What difference does it make whether Actor and Critic share the same network or not?
I'm learning about Actor-Critic reinforcement learning algorithms. One source I encountered mentioned that Actor and Critic can either share one network (but use different output layers) or they can ...