Questions tagged [actor-critic-methods]
For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).
117
questions
-1
votes
0
answers
18
views
Can someone include a link to an article that can help me learn soft actor-critic?
I've been looking through articles on Medium and just papers in general. I can't seem to find anything that really makes sense to me. I always learn better from videos, but there are barely any videos ...
- 109
0
votes
0
answers
27
views
actor critic gradient inconsistent with mathematical proofs
From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to $\theta$ with baseline is proved as:
$$\bigtriangledown_\theta \eta(\...
- 1
1
vote
1
answer
44
views
Reinforcement Learning - Why is actor loss in Actor-Critic multiplied by -1?
Reading through the TensorFlow guide for Actor-Critic learning, I saw that the actor loss is multiplied by -1 when calculating:
The guide says this is to maximize the probabilities of actions with ...
- 13
0
votes
1
answer
49
views
Getting always the same action on an A2C from stable_baselines3
I'm quite new to RL and have been trying to train an A2C model from stable_baselines3 to derive an integer sequence based on 3 other input sequences of floats. I have a custom gym environment that ...
- 121
1
vote
1
answer
48
views
Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
- 41
0
votes
0
answers
32
views
How are Target Values Generated in Alpha Zero Architecture
I am a little confused as to how the target values are generated to train the neural network with the Alpha Zero architecture(in specific to a chess game). I understand how the improved policy is ...
1
vote
0
answers
40
views
Role of $f$ Target Network in DDPG
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
0
votes
1
answer
63
views
Should Cart Pole be immune to Reward Shifting?
I’ve been toying around with cart pole swingup, and have been flummoxed by a problem related to reward shifts. Traditionally, cart pole gives you positive reward if you’re stable and zero reward if ...
- 111
2
votes
1
answer
222
views
Why is soft actor critic an off policy scheme?
I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy ...
- 35
1
vote
1
answer
195
views
Why is Soft Q Learning not an Actor Critic method?
I've been reading these two papers from Haarnoja et. al.:
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
Learning with a Stochastic Actor
Reinforcement Learning with Deep Energy-...
1
vote
1
answer
50
views
Can action be dominated by state features in actor-critic algorithms?
I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I ...
- 331
0
votes
0
answers
121
views
Why do Soft Actor-Critic with automatic temperature tuning use only a single dual variable?
In section 5 of the paper “Soft Actor Critic Algorithms and Applications”, the authors propose to optimize the policy subject to the constraints that the entropy of action distribution should be ...
- 203
1
vote
0
answers
90
views
Has anyone here tried to implement MADDPG for a different environment and succeeded?
Has anyone tried implementing the multi-agent RL algorithm MADDPG (I've linked the paper below)? The paper seems to have a good amount citations, and they do have their code on github. However, a few ...
- 111
3
votes
1
answer
154
views
Why does A2C use the actual returns from an episode in calculating the advantage?
Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?
Basically, why this:
$A(s,a) = \sum_t\gamma^tr_t - V(s)...
- 125
2
votes
1
answer
613
views
Pytorch's Actor-critic implementation seems to be implemented in a Monte-Carlo fashion - why?
In the Actor-Critic example, provided by PyTorch, it seems that the update rule only occurs when the episode ends (like in a Monte-Carlo process). Specifically, in their ...
- 371
1
vote
1
answer
257
views
How the Critic is used to train the Actor in Actor-Critic network
I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.
But I don't fully understand how the critic output (...
- 111
2
votes
1
answer
193
views
What makes TRPO an actor-critic method? Where is the critic?
From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
- 309
0
votes
0
answers
53
views
Why don't approximate TD target use a sample mean?
In practice, we prefer to fit the action-value function by minimizing the Mean Squared Error (MSE) between a parameterized function $Q_{w}(s, a)$ and one-step td target,
\begin{equation}
y = r(s, a) ...
- 41
0
votes
1
answer
64
views
Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?
My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
- 41
1
vote
0
answers
46
views
Linear Actor Critic for continuing task and 1 continuous action => Any comment?
I wish to implement an Actor Critic agent using linear functions for a continuing task with one continuous action. Below the resulting pseudo-code I have reached by my own (the initialization part is ...
- 11
0
votes
0
answers
1k
views
Tensorflow-gpu and multiprocessing
I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds ...
0
votes
0
answers
37
views
As someone starting out in RL, could you help me understand the differences between actor-only, critic-only, and actor-critic methods?
I have been reading some medium articles and these three methods pop up a lot. I am wondering what the differences between these are, what are the advantages of one over the other, etc. Also from my ...
- 101
3
votes
2
answers
251
views
How do we estimate the value of a stochastic policy?
I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
1
vote
1
answer
360
views
How to implement PPO without using a Critic
I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
- 53
1
vote
0
answers
105
views
DDPG agent with convolutional layers for feature extraction [closed]
I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I ...
0
votes
1
answer
120
views
Replay buffer action range in DDPG
I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-...
- 13
1
vote
0
answers
225
views
Why is an action-independent baseline required to reduce variance?
I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
- 49
1
vote
1
answer
773
views
Does Value Loss in Actor Critic not decrease at all?
I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://...
1
vote
0
answers
155
views
How to use Actor-Critic RL with a categorical, state-dependent action space?
I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
- 360
5
votes
1
answer
945
views
Why isn't a target network used for the critic in on-policy actor-critic methods?
Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow:
$$
V_\pi(s_t)...
- 170
1
vote
1
answer
403
views
How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
- 141
2
votes
1
answer
228
views
Joined vs Separate optimizer for Actor-Critic
Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
- 165
1
vote
0
answers
58
views
Setting initial values in DDPG to favor better actions
I'm working on a problem using DDPG.
Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
- 11
1
vote
2
answers
2k
views
PPO when does the update happen?
In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
- 223
0
votes
0
answers
516
views
RLlib's Multi-agent PPO continuous actions turn into nan
After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
- 223
3
votes
0
answers
1k
views
How to interpret the training loss curves in Soft-Actor-Critic (SAC)?
I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
- 45
0
votes
0
answers
33
views
How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?
I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
3
votes
0
answers
327
views
How to deal with a moving target in the Lunar Lander environment with DDPG?
I have noticed that DDPG does rather well at solving environments with a static target.
For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
- 131
1
vote
0
answers
71
views
Understanding advantage estimator in proximal policy optimization
I was reading Proximal Policy Optimization paper. It states following:
The advantage estimator used is:
$\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
- 221
0
votes
1
answer
306
views
Can I train a DQN on the same dataset for multiple epochs?
I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a ...
- 151
0
votes
1
answer
378
views
Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)
I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action.
$Q(s,...
3
votes
0
answers
225
views
How to implement REINFORCE with eligibility traces?
The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
6
votes
1
answer
943
views
What are the advantages of RL with actor-critic methods over actor-only methods?
In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?
This is not a comparison with the Q-learning series, but probably a method of learning ...
- 111
1
vote
0
answers
67
views
Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?
I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard.
My ...
- 26
5
votes
1
answer
2k
views
What difference does it make whether Actor and Critic share the same network or not?
I'm learning about Actor-Critic reinforcement learning algorithms. One source I encountered mentioned that Actor and Critic can either share one network (but use different output layers) or they can ...
- 753
1
vote
1
answer
232
views
Advantage Actor Critic model implementation with Tensorflowjs
I am trying to implement an Actor Critic method that controls an RC car. For this I have implemented a simulated environment and actor critic tensorflowjs models.
My intention is to train a model to ...
- 121
5
votes
1
answer
416
views
Is reinforcement learning only about determining the value function?
I started reading some reinforcement learning literature, and it seems to me that all approaches to solving reinforcement learning problems are about finding the value function (state-value function ...
- 287
0
votes
0
answers
126
views
Variance of the Gaussian policy is not decreasing while training the agent using Soft Actor-Critic method
I've written my own version of SAC(v2) for a problem with continuous action space. While training, the losses for the value network and both q functions steadily decrease down to 0.02-0.03. The loss ...
1
vote
0
answers
129
views
How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?
For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer ...
- 111
0
votes
1
answer
192
views
Why would the reward of A3C with LSTM suddenly drop off after many episodes?
I am training an A3C with stacked LSTM.
During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
- 121