Questions tagged [actor-critic-methods]

For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).

Filter by
Sorted by
Tagged with
3
votes
1answer
99 views

Is reinforcement learning only about determining the value function?

I started reading some reinforcement learning literature, and it seems to me that all approaches to solving reinforcement learning problems are about finding the value function (state-value function ...
2
votes
0answers
29 views

Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: \begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} ...
0
votes
0answers
38 views

Variance of the Gaussian policy is not decreasing while training the agent using Soft Actor-Critic method

I've written my own version of SAC(v2) for a problem with continuous action space. While training, the losses for the value network and both q functions steadily decrease down to 0.02-0.03. The loss ...
1
vote
0answers
38 views

How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?

For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer ...
0
votes
1answer
21 views

Why would the reward of A3C with LSTM suddenly drop off after many episodes?

I am training an A3C with stacked LSTM. During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
0
votes
1answer
128 views

How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?

I'm working in A2C and I have an environment where there is increasing or decreasing in the number of agents. The action space in the environment will not change but the state will change when new ...
0
votes
0answers
52 views

How to modify the Actor-Critic policy gradient algorithm to perform Safe exploration in Reinforcement Learning

I am trying to implement safe exploration technique in [Ref.1]. I am using Soft Actor-Critic algorithm to teach an agent to introduce a bias between 0 and 1 to a specific state of interest in my ...
0
votes
1answer
24 views

Learning only using off-policy samples

When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a ...
1
vote
0answers
21 views

When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?

I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this. Now, I want to know that for a problem/game like Breakout, where ...
1
vote
0answers
27 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
0
votes
0answers
66 views

How to handle actor-critic when actor consistently overestimates one action?

Following multiple guides for actor-critic, I have attempted to implement a single agent one-step version to see how well it would perform on Pong. My understanding is that the actor provides ...
1
vote
0answers
37 views

Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case? I am a bit confused about the off-policy case (right figure). ...
0
votes
1answer
53 views

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be. Is it a scalar that I have to use to update all the weights (so all ...
0
votes
0answers
47 views

Using deep deterministic policy gradient in OpenAI Gym to solve problems with continuous actions

I am trying to do the following: Install the OpenAI baseline algorithms from the following GitHub repository: github.com/openai/baselines by following the instructions in the readme file. Train an ...
1
vote
1answer
46 views

Why is the “reward to go” replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...
0
votes
1answer
115 views

Why is my Soft Actor-Critic's policy and value function losses not converging?

I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging,...
1
vote
1answer
203 views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
1
vote
0answers
56 views

Actor-Critic implementation not learning

I've implemented a vanilla actor-critic and have run into a wall. My model does not seem to be learning the optimal policy. The red graph below shows its performance in cartpole, where the algorithm ...
2
votes
1answer
59 views

Learning policy where action involves discrete and continuous parameters

Typically it seems like reinforcement learning involves learning over either a discrete or a continuous action space. An example might be choosing from a set of pre-defined game actions in Gym Retro ...
2
votes
1answer
305 views

Why isn't my implementation of A2C for the the atari pong game converging?

I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different. https://colab.research.google.com/drive/...
2
votes
2answers
181 views

Advantage computed the wrong way?

Here is the code written by Maxim Lapan. I am reading his book (Deep Reinforcement Learning Hands-on). I have seen a line in his code which is really weird. In the accumulation of the policy gradient $...
4
votes
1answer
85 views

Would you categorize policy iteration as an actor-critic reinforcement learning approach?

One way of understanding the difference between value function approaches, policy approaches and actor-critic approaches in reinforcement learning is the following: A critic explicitly models a value ...
3
votes
1answer
107 views

How does being on-policy prevent us from using the replay buffer with the policy gradients?

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we discussed in ...
1
vote
1answer
46 views

Why a single trajectory can be used to update the policy network $\theta$ in A3C?

The Deep RL bootcamp on policy gradient techniques gives the update equation for the policy network in A3C as $\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}...
1
vote
1answer
121 views

Once the environments are vectorized, how do I have to gather immediate experiences for the agent?

My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the ...
1
vote
0answers
60 views

Is the reward following after time step $t+1$ collected based on current policy?

I am currently learning policy gradient methods from the Deep RL boot camp by Pieter Abbeel in which he explains the actor-critic algorithm derivation. At around minute 39, he explains that the sum ...
1
vote
0answers
55 views

How should I deal with variable batch size in A3C?

I am fairly new to reinforcement learning (RL) and deep RL. I have been trying to create my first agent (using A3C) that selects an optimal path with the reward being some associated completion time (...
3
votes
1answer
137 views

What is the advantage of using more than one environment with the advantage actor-critic?

make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4")) envs = [make_env() for _ in range(NUM_ENVS)] Here is a code you can look at. ...
2
votes
0answers
49 views

Original source of the TD Advantage Actor-Critic algorithm?

What is the original source of the TD Advantage Actor-Critic algorithm? I found the following tutorial really helpful for learning the algorithm: https://medium.com/@asteinbach/actor-critic-using-...
0
votes
0answers
17 views

Won't the copy of the weights of the worker model to the global model erase the work of other workers in A3C?

I was reading the article Deep Reinforcement Learning: Playing CartPole through Asynchronous Advantage Actor-Critic (A3C) with tf.keras and eager execution. From my understanding, we copy the weights ...
2
votes
0answers
48 views

What is the gradient of the Q function with respect to the policy's parameters?

I have been recently studying Actor-Critic algorithms, and I ran into the following question. Let $Q_{\omega}$ be the critic network, and $\pi_{\theta}$ be the actor. It is known that in order to ...
1
vote
1answer
91 views

What does the notation $\partial \theta_{\pi}$ mean in this actor-critic update rule?

One of the steps in the actor-critic algorithm is $$\partial \theta_{\pi} \gets \partial \theta_{\pi} + \nabla_{\theta}\log\pi_{\theta} (a_i | s_i) (R - V_{\theta}(s_i))$$ For me, $\theta$ are just ...
0
votes
0answers
25 views

Gradients null for actor in Acton-Value actor-critic

I'm trying to implement a simple actor-critic algorithm based on action-value critic. I'm following the pseudocode from David Silver (please find attached his slide) ...
2
votes
1answer
142 views

How to set the target for the actor in A2C?

I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, ...
4
votes
1answer
122 views

Why not more TD(𝜆) in actor-critic algorithms?

Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for ...
1
vote
0answers
72 views

Implementing Actor-Critic with Experience Replay for Continuous Action Spaces

I have been trying to implement the ACER algorithm for continuous action spaces in reinforcement learning. The paper for the algorithm can be found here: Sample Efficient Actor-Critic with Experience ...
1
vote
0answers
23 views

Can I apply experience on naive actor critic directly? Should it work?

Can I apply experience replay on naive actor-critic directly? Should it work? I have tried that but unfortunately it didn't work.
2
votes
0answers
95 views

How does the update rule for the one-step actor-critic method work?

Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggests an ...
1
vote
0answers
55 views

Why I got the same action when I train A2C when I increase the number of episodes?

I'm working on an advantage actor-critic (A2C) reinforcement learning model but the problem when I trained the system for 3500 episodes, I start to get the same action for all my testing results. ...
4
votes
1answer
84 views

What is the intuition behind the TD(0) equation with average reward, and how is it derived?

In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10): $$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{...
3
votes
1answer
131 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
1
vote
0answers
128 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
3
votes
1answer
95 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
0
votes
1answer
256 views

How can the derivative of a neural network be calculated, given no mathematical expression?

Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a ...
1
vote
1answer
474 views

How is the actor-critic algorithm guaranteed to converge?

From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function ...
1
vote
1answer
489 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
1
vote
0answers
77 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
1
vote
1answer
198 views

A2C Critic Loss Interpretation

I'm working on an Advantage A2C implementation, and I just finished creating the value network $\hat{V_{\phi}}$. I train this network with the standard MSE loss of discounted rewards-to-go:$$\|\hat{V_{...
7
votes
1answer
174 views

What is the purpose of the actor in actor-critic algorithms?

For discrete action spaces, what is the purpose of the actor in actor-critic algorithms? My current understanding is that the critic estimates the future reward given an action, so why not just take ...