# Questions tagged [actor-critic-methods]

For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).

124 questions
Filter by
Sorted by
Tagged with
1 vote
30 views

### DDPG model outputting a fixed action at every timestep

I am trying to create a Car Following model, for which i am using DDPG. My action is acceleration bounded in a range of [-3,3] m/s2. While training the model, for every state it gives a single ...
13 views

### Soft Actor Critic policy update depending on q function in discrete action space

Reference: https://spinningup.openai.com/en/latest/algorithms/sac.html In the psuedo-code of the algorithm, line 14, the actor update is written as to maximize the q-function. Theoretically, this ...
• 133
140 views

### Notation used in paper on Continuous Time Reinforcement Learning

I am working on implementing the learning shown in this paper: https://homes.cs.washington.edu/~todorov/courses/amath579/reading/Continuous.pdf In the paper, the authors devise a continuous time ...
36 views

### Non differentiable loss function train with actor critic style

I'm working on a project where a non differentiable loss is there. I'm thinking about how should I deal with them. My model is a very big lstm model (about 1M parameter), and after 500 steps (not sure ...
• 13
1k views

### Is PPO a policy-based method or an actor-critique-based method?

as far as i understand there are 3 categories of Reinforcement algorithms: Value-based methods (like DQN or Sarsa) Policy-based methods (like REINFORCE) Actor-critic-based methods (like A2C) To ...
• 256
63 views

### How and whether to apply Reinforcement Learning in an Environment with a precise and always available Evaluation?

Say we want to train an agent $A$ in an environment $E$ which provides a continuous loss $L$. That is, we want $A$ to choose its actions $a$ so that it minimizes the mistake it does, i.e., it ...
• 153
83 views

### In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?

Taking as an example the Advantage Actor Critic, the objective function is: \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\...
102 views

### Why is my agent stuck on the same action in my Twin Delayed Deep Deterministic Policy Gradient (TD3) program?

I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to ...
248 views

### Unable to interpret DDPG actor-critic loss curves

I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
30 views

• 170
1 vote
539 views

### How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?

Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
• 151
329 views

### Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
• 165
1 vote
63 views

### Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
• 11
3k views

### PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
• 243
612 views

### RLlib's Multi-agent PPO continuous actions turn into nan

After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
• 243
2k views

### How to interpret the training loss curves in Soft-Actor-Critic (SAC)?

I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
• 45
34 views

### How can I compare the results of AC1 with the results of A3C (on the CartPole environment)?

I am implementing A3C for the CartPole environment. I want to compare the results I got from A3C with the ones I got from AC1. The problem is I don't know which process to look at. If I use, let's say,...
445 views

### How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
• 131
1 vote
117 views