Questions tagged [actor-critic-methods]

For questions related to the family of reinforcement learning algorithms denoted by "actor-critic", where there is an actor (a policy) and a critic (a value function).

Filter by
Sorted by
Tagged with
4
votes
1answer
119 views

What is the intuition behind the TD(0) equation with average reward, and how is it derived?

In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10): $$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{...
3
votes
1answer
177 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
1
vote
0answers
178 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
4
votes
1answer
118 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
1
vote
1answer
578 views

How can the derivative of a neural network be calculated, given no mathematical expression?

Neural networks (NNs) are used as approximators in reinforcement learning (RL). To update the policy in RL, the actor network's gradients w.r.t its weights are needed. Since NN doesn't have a ...
1
vote
1answer
660 views

How is the actor-critic algorithm guaranteed to converge?

From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function ...
1
vote
1answer
638 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
1
vote
0answers
107 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
1
vote
1answer
276 views

A2C Critic Loss Interpretation

I'm working on an Advantage A2C implementation, and I just finished creating the value network $\hat{V_{\phi}}$. I train this network with the standard MSE loss of discounted rewards-to-go:$$\|\hat{V_{...
7
votes
1answer
242 views

What is the purpose of the actor in actor-critic algorithms?

For discrete action spaces, what is the purpose of the actor in actor-critic algorithms? My current understanding is that the critic estimates the future reward given an action, so why not just take ...
1
vote
0answers
328 views

Do we need to use the experience replay buffer with the A3C algorithm?

I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm. The approached I used is the following: ...
3
votes
0answers
180 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
4
votes
2answers
207 views

How is parallelism implemented in RL algorithms like PPO?

There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory ...
1
vote
1answer
69 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
1
vote
0answers
175 views

Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network

I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system. the state space and the ...
2
votes
1answer
283 views

What is the difference between A2C and running an agent in an environment vector?

I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector? I personally ...
1
vote
0answers
30 views

Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?

According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
3
votes
1answer
115 views

Can A3C update the policy / critic on a local machine without needing to copy?

To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/...
0
votes
1answer
799 views

Getting NaN from A3C PPO model [closed]

I've pieced together this A3C w/ PPO Gym Pendulum example, but I'm finding after a while, when attempting to get the action from the model, I get a NaN return: ...
2
votes
1answer
133 views

Why is gradient ascent necessary when training Actor Critic agents?

I have read a lot on Actor Critic and I'm not convinced that there is a qualitative difference doing direct gradient updates on the network and slightly adjusting a soft-max output in the direction of ...
1
vote
0answers
244 views

How the actor use the output from the critic to make action in actor-critic network?

I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network. Below you have the most popular ...
1
vote
1answer
120 views

What is the most biologically plausible representation for the actor and critic?

Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either mutually exclusive with each other (e.g., go ...
2
votes
1answer
478 views

Are A2C or A3C suitable for episodic tasks where the reward is delivered only at the end of the episode?

My understanding of the main idea behind A2C / A3C is that we run small segments of an episode to estimate the return using a trainable value function to compensate for the unseen final steps of the ...
2
votes
2answers
208 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{s_t\...
2
votes
1answer
449 views

In online one step actor critic, why does the weights update become less significant as the episode progresses?

The Reinforcement Learning Book by Richard Sutton et al, section 13.5 shows an online actor critic algorithm. Why do the weights updates depend on the discount factor via $I$? It seems that the more ...
3
votes
1answer
494 views

Meaning of Actor Output in Actor Critic Reinforcement Learning

In actor critic, The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by ...
0
votes
1answer
474 views

Why is this PyTorch implementation of the actor-critic algorithm inconsistent with the mathematical formulas?

This PyTorch implementation of the actor-critic algorithm calculates the losses like so: ...
2
votes
1answer
150 views

Why is the actor-critic algorithm limited to using on-policy data?

Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
2
votes
1answer
607 views

How do I calculate the policy in the Proximal Policy Optimization algorithm?

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are ...
6
votes
2answers
1k views

What are the available exploration strategies for continuous action space scenarios in RL?

I'm building a deep neural network to serve as the policy estimator in an actor-critic reinforcement learning algorithm for a continuing (not episodic) case. I'm trying to determine how to explore ...
13
votes
3answers
7k views

What is the difference between actor-critic and advantage actor-critic?

I'm struggling to understand the difference between actor-critic and advantage actor-critic. At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an ...
1
vote
1answer
130 views

Can neuro-evolution methods be combined with A3C?

As a amateur researcher and tinkerer, I've been reading up on neuro-evolution networks (e.g. NEAT) as well as the A3C RL approach presented by Mnih et al and got to wondering if anyone has ...
6
votes
2answers
1k views

What is the difference between on and off-policy deterministic actor-critic?

In the paper Deterministic Policy Gradient Algorithms, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic". I don't know what's the difference ...
4
votes
1answer
253 views

Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm

I have a general question about the updating of the network/model in the PPO algorithm. If I understand it correctly, there are multiple iterations of weight updates done on the model with data that ...
3
votes
1answer
1k views

Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game?

I implemented the Actor-Critic with n-step TD prediction to learn to play the 2048 game For the environment, I don't use this 2048 implementation. I use a simple one without any graphical interface, ...
30
votes
4answers
10k views

How should I handle invalid actions (when using REINFORCE)?

I want to create an AI which can play five-in-a-row/gomoku. I want to use reinforcement learning for this. I use policy gradient method, namely REINFORCE, with baseline. For the value and policy ...

1
2