Questions tagged [policy-gradients]

For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

Filter by
Sorted by
Tagged with
0 votes
1 answer
24 views

Does the policy search work if there is no state to state dependency through actions?

There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
user avatar
0 votes
0 answers
23 views

Can DDPG algorithm obtain time-dependent and time-independent actions simultaneously?

I am new to Reinforcement Learning. I have been working on a problem using Deep Deterministic Policy Gradient (DDPG). I would like to know if it is possible to apply this algorithm to an optimization ...
user avatar
1 vote
0 answers
48 views

Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
user avatar
  • 49
0 votes
1 answer
64 views

Does importance sampling really improve sampling efficiency of policy gradient methods such as TRPO or PPO?

Vanilla policy gradient has a loss function: $\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$ while in TRPO it is: $\mathcal{L}_{\pi_{\...
user avatar
0 votes
2 answers
81 views

Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
user avatar
  • 746
0 votes
0 answers
18 views

How to reduce the variance of stochastic policy gradient for continuous actions in a partially observable environment?

I am trying to implement a stochastic policy gradient for continuous actions in a partially observable CartPole environment. Specifically, only the current cart position and pole angle are visible, ...
user avatar
  • 173
1 vote
1 answer
55 views

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

Consider the following algorithm from the textbook titled Reinforcement Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart While playing the game for the generation of ...
user avatar
  • 2,977
2 votes
0 answers
46 views

In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?

In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated actions [may be] chosen from a normal (Gaussian) distribution. However, I ...
user avatar
1 vote
0 answers
37 views

Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay

In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration like this : ...
user avatar
1 vote
1 answer
30 views

How to mix grid matrix and explicit values when designing RL state?

I'm trying to do multi-agent reinforcement learning on the grid world navigation task where multiple agents try to collectively reach multiple goals while avoiding collisions with stationary obstacles ...
user avatar
1 vote
1 answer
162 views

What is a policy training target in AlphaZero?

In AlphaZero's attached pseudocode, they create a training target for the policy network in this way. ...
user avatar
  • 159
0 votes
0 answers
27 views

Does it make sense to use the online network for making moves for both players in PPO for 2-player games?

I want to use PPO (or TRPO) to learn two-player games like 4 connect. Now, I'm thinking about whether or not it makes sense to use the online network for making moves for both players, in order to ...
user avatar
2 votes
2 answers
94 views

What specifically is the gradient of the log of the probability in policy gradient methods?

I am getting tripped up slightly by how specifically the gradient is calculated in policy gradient methods (just the intuitive understanding of it). This Math Stack Exchange post is close, but I'm ...
user avatar
0 votes
0 answers
56 views

FrozenLake-v0 not training using REINFORCE

I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all. I have used the same neural architecture ...
user avatar
  • 101
3 votes
1 answer
110 views

Why exactly was previously believed that the deterministic policy gradient did not exist?

I'm reading the paper Deterministic Policy Gradient Algorithms, David Silver et al. First of all, in the introduction, the author says that It was previously believed that the deterministic policy ...
user avatar
  • 133
0 votes
0 answers
84 views

Why does this PPO agent doesn't learn at all?

I am trying to code a Proximal Policy Optimization algorithm in Pytorch by myself for the OpenAI gym preduum-v0 environment. However, I find my learning curve is ...
user avatar
1 vote
0 answers
22 views

Can we learn a policy network via a sequence of manually determined actions?

In policy gradients, is it possible to learn the policy if the chain of actions is selected and performed manually/externally (e.g. by myself or by someone else who I have no influence over)? For ...
user avatar
  • 121
0 votes
1 answer
68 views

Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o_t$) with a camera at each location (action $...
user avatar
0 votes
0 answers
21 views

multi agent deep deterministic policy gradient for discrete actions

I am solving a multi agent problem where each agent has a critic and actor. The problem I am solving has discrete actions and discrete states. I came cross multi-agent deep deterministic policy ...
user avatar
0 votes
0 answers
90 views

DDPG or PPO don't work well with my custom non gym environment

I have a project to control a robot with right and left wheel speeds, and my step time is not constant. Because my outputs are continuous (right wheel speed, left wheel speed, and time step), I try to ...
user avatar
  • 3
1 vote
1 answer
51 views

How to interpret the policy gradient expression in reinforcement learning?

I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ...
user avatar
2 votes
1 answer
89 views

Why adding a baseline doesn't affect the policy gradient?

On the OpenAI's Spinning Up, they justify the fact that adding a baseline $b(s_t)$ in the policy gradient doesn't change its gradient by saying that this is an immediate consequence of the EGLP Lemma ...
user avatar
0 votes
0 answers
21 views

Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space

I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient. 1, in my question, each agent has its own action space. When doing the sampling, for each ...
user avatar
1 vote
1 answer
87 views

How does the neural network learn when used in the REINFORCE algorithm?

As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just a single loss value. How does the neural network learn to differentiate between good ...
user avatar
0 votes
0 answers
35 views

A2C value function outputs keep increasing

I was implementing the A2C algorithm with as close to baseline setup as possible, and this is the code I came up with. The problem is that even after multiple rechecks, the program isn't showing ...
user avatar
  • 51
0 votes
0 answers
17 views

How do I quantify the difference in sample efficiency for two almost similar methods?

I am comparing my coded TD3 (Twin-Delayed DDPG) and the same TD3 (same hyperparameters) but with Priority Replay Buffer instead of a normal Replay Buffer. From what I have read, PER (Priority ...
user avatar
0 votes
1 answer
52 views

How to formulate discounted return in cartpole?

I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return: finite horizon undiscounted ...
user avatar
4 votes
1 answer
997 views

Are policy gradient methods good for large discrete action spaces?

I have seen this question asked primarily in the context of continuous action spaces. I have a large action space (~2-4k discrete actions) for my custom environment that I cannot reduce down further: ...
user avatar
-2 votes
1 answer
172 views

Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions

i'm trying to solve a problem in which i need to carry out reinforcement learning with multiple simultaneous actions in continuous action space . i checked the multiagent structure; however, im trying ...
user avatar
6 votes
1 answer
132 views

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being \begin{equation} \mu(s) = \frac{\...
user avatar
1 vote
0 answers
348 views

PPO in continuous control not working

I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
user avatar
1 vote
1 answer
60 views

Is the policy gradient expression in Fundamentals of Deep Learning wrong?

I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning. Here is the whole paragraph: Policy Learning via Policy ...
user avatar
1 vote
0 answers
25 views

Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?

I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL ...
user avatar
1 vote
1 answer
144 views

Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?

I read the book "Foundation of Deep Reinforcement Learning, Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function: $$ J\left(\...
user avatar
1 vote
0 answers
23 views

Understanding advantage estimator in proximal policy optimization

I was reading Proximal Policy Optimization paper. It states following: The advantage estimator used is: $\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
user avatar
  • 211
0 votes
1 answer
115 views

How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?

In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation: \begin{align} \nabla_{\theta}J &\propto \sum_s \mu(s) \...
user avatar
1 vote
1 answer
24 views

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have ...
user avatar
  • 11
1 vote
1 answer
76 views

Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy. Lillicrap et al. denote the policy gradient by $$\...
user avatar
  • 35
1 vote
1 answer
121 views

It is mathematically correct to use a Policy Gradient method for 1-step trajectories?

I have come across a Google paper that uses the REINFORCE algorithm (a Policy Gradient Method) for a case where the trajectory of the episodes it proposes would be ...
user avatar
  • 171
1 vote
0 answers
73 views

Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)

I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of ...
user avatar
1 vote
0 answers
50 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...
user avatar
2 votes
0 answers
72 views

$\nabla \log \pi$ with respect to some parameters constantly being zero

I am new to reinforcement learning. May I ask a simple (and maybe a bit silly) question here? I am trying to use the "one-step actor-critic" method to train a robot on a gridworld. Let's ...
user avatar
  • 21
1 vote
0 answers
30 views

Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?

I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard. My ...
user avatar
  • 26
1 vote
0 answers
281 views

Designing Policy-Network for Deep-RL with Large, Variable Action Space

I am attempting a project involving training an agent to play a game using deep reinforcement learning. This project has a few features that complicate the design of the neural network: The action ...
user avatar
2 votes
0 answers
141 views

Policy gradient: Does it use the Markov property?

To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see spinningup tutorial): $$ \begin{align} P_\theta(\tau) &= P_\theta(s_0, a_0, ...
user avatar
  • 121
1 vote
0 answers
166 views

Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
user avatar
  • 31
2 votes
0 answers
35 views

Understanding neural network achitectures in policy gradient reinforcement learning for continuous state and action space

I am trying to train a neural network using reinforcement learning / policy gradient methods. The states, i.e. the inputs, as well as the actions I am trying to sample are vectors with each element ...
user avatar
3 votes
0 answers
230 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
user avatar
2 votes
0 answers
59 views

Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: \begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} ...
user avatar
  • 337
1 vote
0 answers
59 views

What are the disadvantages of actor-only methods with respect to value-based ones?

While the advantages of actor-only algorithms, the ones that search directly the policy without the use of the value function, are clear (possibility of having a continuous action space, a stochastic ...
user avatar
  • 281