# Questions tagged [actor-critic-methods]

For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).

98 questions
Filter by
Sorted by
Tagged with
19 views

### Is there a multi-task RL algorithm that supports different action spaces for each agent?

I'm currently working on a project in which I need apply multi-task reinforcement learning. Over the same state space, each agent aims to do a separate task, but the action spaces of agents are ...
• 1
1 vote
47 views

### Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
• 49
60 views

### Does Value Loss in Actor Critic not decrease at all?

I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://...
1 vote
51 views

### How to use Actor-Critic RL with a categorical, state-dependent action space?

I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
• 266
679 views

### Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow:  V_\pi(s_t)...
• 125
1 vote
72 views

### How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?

Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
• 121
21 views

### multi agent deep deterministic policy gradient for discrete actions

I am solving a multi agent problem where each agent has a critic and actor. The problem I am solving has discrete actions and discrete states. I came cross multi-agent deep deterministic policy ...
1 vote
60 views

### Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
• 143
1 vote
34 views

### Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
• 11
1 vote
224 views

### PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
• 203
29 views

### How to train critic network in PPO with multiple actor?

I try to code an algorithm to control a robot's movement with continuous action space. I use this question and create an actor network for each action.How can policy gradients be applied in the case ...
53 views

### tanh activation function output is not between -1 and 1 for continuous action PPO

I am using RLlib's (Ray = 1.4.0) PPO policy, and my first layer after the input (Conv layer) is producing a strange output keeping in mind that the activation for the output is a tanh, which I do ...
• 203
215 views

### RLlib's Multi-agent PPO continuous actions turn into nan

After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
• 203
43 views

### Actor-critic reinforcement learning updates and episode length

I am currently using a TD3 agent-critic network to control a vehicle suspension system, where the reward (or rather a penalty) is based on the vertical acceleration of the mass and is calculated at ...
273 views

### How to interpret the training loss curves in Soft-Actor-Critic (SAC)?

I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
• 35
24 views

### How to obtain STD from Neural Network with 2 continuous action output

In my Environment, I have two continuous action space self.action_space = spaces.Box(low=np.array([0.,0.]), high=np.array([4.,0.02]), shape=(2,), dtype=np.float32) ...
29 views

### How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?

I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
149 views

### How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
• 131
1 vote
23 views

77 views

• 161
409 views

### Would you categorize policy iteration as an actor-critic reinforcement learning approach?

One way of understanding the difference between value function approaches, policy approaches and actor-critic approaches in reinforcement learning is the following: A critic explicitly models a value ...
• 81
390 views

### How does being on-policy prevent us from using the replay buffer with the policy gradients?

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we discussed in ...
• 161
56 views

### Why a single trajectory can be used to update the policy network $\theta$ in A3C?

The Deep RL bootcamp on policy gradient techniques gives the update equation for the policy network in A3C as \$\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}...
• 1,201
1 vote