A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [actor-critic]

For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

How does the update rule for the one-step actor-critic method work?

Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggest an ...
1
vote
0answers
25 views

Why I got the same action when I train A2C when I increase the number of episodes?

I'm working on an advantage actor-critic (A2C) reinforcement learning model but the problem when I trained the system for 3500 episodes, I start to get the same action for all my testing results. ...
3
votes
1answer
55 views

What is the intuition behind the TD(0) equation with average reward, and how is it derived?

In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10): $$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{...
1
vote
1answer
52 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
1
vote
0answers
17 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
3
votes
1answer
35 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
0
votes
1answer
73 views

How can the derivative of a neural network be calculated, given no mathematical expression?

Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a ...
1
vote
1answer
96 views

How is the actor-critic algorithm guaranteed to converge?

From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function ...
1
vote
0answers
145 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
1
vote
0answers
41 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
0
votes
1answer
87 views

A2C Critic Loss Interpretation

I'm working on an Advantage A2C implementation, and I just finished creating the value network $\hat{V_{\phi}}$. I train this network with the standard MSE loss of discounted rewards-to-go:$$\|\hat{V_{...
6
votes
1answer
82 views

Purpose of Actor in Actor-Critic algorithm?

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? My current understanding is that the critic estimates the future reward given an action, so why not just take ...
1
vote
0answers
101 views

Do we need to use the experience replay buffer with the A3C algorithm?

I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm. The approached I used is the following: ...
3
votes
0answers
49 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
1
vote
1answer
39 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
1
vote
0answers
74 views

Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network

I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system. the state space and the ...
1
vote
0answers
29 views

Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?

According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
3
votes
1answer
80 views

Can A3C update the policy / critic on a local machine without needing to copy?

To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/...
0
votes
1answer
373 views

Getting NaN from A3C PPO model [closed]

I've pieced together this A3C w/ PPO Gym Pendulum example, but I'm finding after a while, when attempting to get the action from the model, I get a NaN return: ...
2
votes
1answer
91 views

Why is gradient ascent necessary when training Actor Critic agents?

I have read a lot on Actor Critic and I'm not convinced that there is a qualitative difference doing direct gradient updates on the network and slightly adjusting a soft-max output in the direction of ...
1
vote
0answers
103 views

How the actor use the output from the critic to make action in actor-critic network?

I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network. Below you have the most popular ...
0
votes
1answer
108 views

What is the most biologically plausible representation for the actor and critic?

Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either mutually exclusive with each other (e.g., go ...
2
votes
1answer
320 views

Are A2C or A3C suitable for episodic tasks where the reward is delivered only at the end of the episode?

My understanding of the main idea behind A2C / A3C is that we run small segments of an episode to estimate the return using a trainable value function to compensate for the unseen final steps of the ...
0
votes
0answers
103 views

How to correctly discount actor critic with experience replay?

In my related question, I asked about the one step actor critic from The Reinforcement Learning Book by Richard Sutton et al, section 13.5: The learning is becoming less significant as the episode ...
2
votes
1answer
141 views

In online one step actor critic, why does the weights update become less significant as the episode progresses?

The Reinforcement Learning Book by Richard Sutton et al, section 13.5 shows an online actor critic algorithm. Why do the weights updates depend on the discount factor via $I$? It seems that the more ...
3
votes
1answer
247 views

Meaning of Actor Output in Actor Critic Reinforcement Learning

In actor critic, The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by ...
8
votes
3answers
2k views

Why does is make sense to normalize rewards per episode in reinforcement learning?

In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so rewards = (rewards - rewards.mean()) / (rewards.std() + eps) on ...
1
vote
1answer
77 views

Why is the actor-critic algorithm limited to using on-policy data?

Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
2
votes
1answer
402 views

How do I calculate the policy in the Proximal Policy Optimization algorithm?

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are ...
9
votes
2answers
4k views

What is the difference between Actor-Critic and Advantage Actor-Critic?

I'm struggling to understand the difference between Actor-Critic and Advantage Actor-Critic. At least I know they are different from Asynchronous Advantage Actor-Critic (A3C), as A3C adds ...
1
vote
1answer
106 views

Can neuro-evolution methods be combined with A3C?

As a amateur researcher and tinkerer, I've been reading up on neuro-evolution networks (e.g. NEAT) as well as the A3C RL approach presented by Mnih et al and got to wondering if anyone has ...
3
votes
2answers
525 views

What is the difference between on and off-policy deterministic actor-critic?

In the paper Deterministic Policy Gradient Algorithms, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic". I don't know what's the difference ...
3
votes
0answers
174 views

Understanding multi iteration update of model in Policy Gradient PPO algorithm

I have a general question about the updating of the network/model in the PPO algorithm. If I understand it correctly, there are multiple iterations of weight updates done on the model with data that ...