Questions tagged [policy-gradients]

For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

Filter by
Sorted by
Tagged with
4
votes
1answer
98 views

What, exactly, does the REINFORCE update equation mean?

I understand that this is the update for the parameters of a policy in REINFORCE: $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t} $$ Where 𝑣𝑡 is ...
3
votes
2answers
69 views

Why does (not) the distribution of states depend on the policy parameters that induce it?

I came across the following proof of what's commonly referred to as the log-derivative trick in policy-gradient algorithms, and I have a question - While transitioning from the first line to the ...
2
votes
0answers
30 views

What's an example of a simple policy but a complex value function?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
2
votes
1answer
25 views

Generation of 'new log probabilities' in continuous action space PPO

I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target ...
4
votes
1answer
77 views

How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?

In section 3 of the paper Continuous control with deep reinforcement learning, the authors write As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & ...
3
votes
1answer
106 views

Why does REINFORCE work at all?

Here's a screenshot of the popular policy-gradient algorithm from Sutton and Barto's book - I understand the mathematical derivation of the update rule - but I'm not able to build intuition as to why ...
1
vote
0answers
28 views

Customized food for persons based on their profile using Reinforcement learning

I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based ...
1
vote
1answer
38 views

DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper Continuous Control using Deep Reinforcement Learning on the MountainCarContinuous-v0 gym environment. I ...
3
votes
1answer
72 views

Comparing the derivation of the Deterministic Policy Gradient Theorem with the standard Policy Gradient Theorem

I would like to understand the difference between the standard policy gradient theorem and the deterministic policy gradient theorem. These two theorem are quite different, although the only ...
0
votes
1answer
65 views

Is it possible to have a fixed trajectory size in the vanilla policy gradient algorithm?

In the concept of the vanilla policy gradient algorithm, is it possible for our trajectory size to be fixed? For example, my environment is the space of embedded images (using a pre-trained encoder to ...
1
vote
1answer
75 views

What is the difference between vanilla policy gradient and advantage actor-critic?

What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)? By vanilla policy gradient I am specifically referring to spinning up's ...
3
votes
1answer
103 views

Choosing a policy improvement algorithm for a continuing problem with continuous action and state-space

I'm trying to decide which policy improvement algorithm to use in the context of my problem. But let me emerge you into the problem Problem I want to move a set of points in a 3D space. Depending on ...
1
vote
1answer
57 views

Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?

I am trying to implement the DDPG algorithm based on this paper. The part that confuses me is the actor network's update. I don't understand why the policy loss is simply the mean of $-Q(s, \mu(s))$, ...
2
votes
1answer
65 views

Is it common to have extreme policy's probabilities?

I have implemented several policy gradient algorithms (REINFORCE, A2C, and PPO) and am finding that the resultant policy's action probability distributions can be rather extreme. As a note, I have ...
1
vote
0answers
53 views

In continuous action spaces, how is the standard deviation, associated with Gaussian distribution from which actions are sampled, represented?

I have a question about implementing policy gradient methods for problems with continuous action spaces. Assume that actions are sampled from a diagonal Gaussian distribution with mean vector $\mu$ ...
2
votes
0answers
47 views

What kind of policy evaluation and policy improvement AlphaGo, AlphaGo Zero and AlphaZero are using

I'm trying to find out what kind of policy improvement and policy evaluation AlphaGo, AlphaGo Zero, and AlphaZero are using. By looking into their respective paper and SI, I can conclude that it is a ...
1
vote
1answer
37 views

How to optimize neural network parameters with REINFORCE

I've seen a few mentions in papers that neural network parameters can be found using REINFORCE algorithm. It was mentioned in the context of nondifferentiable operations involving e.g. step function ...
2
votes
1answer
51 views

How can I classify policy gradient methods in RL?

In the book of Barto and Sutton, there are 3 methods presented that solve an RL problem: DP, Monte Carlo, and TD. But which category does policy gradient methods (or actor-only methods) classify in? ...
2
votes
1answer
44 views

How should we interpret all the different metrics in reinforcement learning?

I'm trying to train some deep RL agents using policy gradient methods like AC and PPO. While training, I have a ton of different metrics being monitored. I understand that the ultimate goal is to ...
0
votes
0answers
29 views

2 out of 7 Observations Defined in MATLAB DDPG Reinforcement Learning Environment. Are the rest given random values?

After reading up one Deep Deterministic Policy Gradient, I found this example on MATLAB: https://www.mathworks.com/help/reinforcement-learning/ug/train-agent-to-control-flying-robot.html#...
4
votes
1answer
76 views

What happens when you select actions using softmax instead of epsilon greedy in DQN?

I understand the two major branches of RL are Q-Learning and Policy Gradient methods. From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
2
votes
2answers
42 views

Understanding the “unroling” step in the proof of the policy gradient theorem

In the proof of the policy gradient theorem in the RL book of Sutton and Barto (that I shamelessly paste here): there is the "unrolling" step that is supposed to be immediately clear With ...
1
vote
0answers
34 views

Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case? I am a bit confused about the off-policy case (right figure). ...
0
votes
1answer
50 views

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be. Is it a scalar that I have to use to update all the weights (so all ...
1
vote
0answers
66 views

What is the proof that “reward-to-go” reduces variance of policy gradient?

I am following the OpenAI's spinning up tutorial Part 3: Intro to Policy Optimization. It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the ...
2
votes
1answer
157 views

Non-differentiable reward function to update a neural network

In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network. In the paper Neural Architecture Search with ...
1
vote
2answers
76 views

Should I use exploration strategy in Policy Gradient algorithms?

In policy gradient algorithms the output is a stochastic policy - a probability for each action. I believe that if I follow the policy (sample an action from the policy) I make use of exploration ...
0
votes
0answers
30 views

Is there any advantage to using a non-diagonal covariance matrix for a policy distribution?

For reinforcement learning implementations with a gym.spaces.Box action space, which is the product of $k$ real closed intervals, it is common (actually more like ...
1
vote
0answers
48 views

Using a model-based method to build an accurate day trading environment model

There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects : Value-based and policy-based On-policy and off-policy Model-free and ...
1
vote
0answers
37 views

How can I perform policy update in python? [closed]

I'm using Python and tensorflow to implement a Deep Q-learning with experience replay in a continous action and state spaces and I have used two neural networks to approximate both the policy function ...
1
vote
0answers
44 views

Policy Gradient on Tic-Tac-Toe not working

I wanted to implement the Policy Gradient on Tic-Tac-Toe. I tried to use the code that worked for any environment like CartPole-v0 to my Tic-Tac-To game. But it is not learning. There are no errors. ...
1
vote
1answer
42 views

In vanilla policy gradient is the baseline lagging behind the policy?

Vanilla policy gradient algorithm (using baseline to reduce variance) acc to here (page 16) Initialize policy parameter θ, baseline b for iteration=1, 2, . . . do Collect a set of ...
2
votes
1answer
54 views

Learning policy where action involves discrete and continuous parameters

Typically it seems like reinforcement learning involves learning over either a discrete or a continuous action space. An example might be choosing from a set of pre-defined game actions in Gym Retro ...
2
votes
1answer
84 views

How do I derive the gradient with respect to the parameters of the softmax policy?

The gradient of the softmax eligibility trace is given by the following: \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{...
2
votes
0answers
69 views

Is this the correct gradient for log of softmax? [duplicate]

I am currently implementing the very basic version (REINFORCE) of the Monte Carlo policy gradient algorithm. I was wondering if this is the correct gradient for the log of softmax. \begin{align} \...
2
votes
2answers
173 views

Advantage computed the wrong way?

Here is the code written by Maxim Lapan. I am reading his book (Deep Reinforcement Learning Hands-on). I have seen a line in his code which is really weird. In the accumulation of the policy gradient $...
3
votes
1answer
80 views

On-policy preventing us from using the replay buffer with the PG?

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we ...
1
vote
1answer
42 views

Why a single trajectory can be used to update the policy network $\theta$ in A3C?

The Deep RL bootcamp on policy gradient techniques gives the update equation for the policy network in A3C as $\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}...
1
vote
2answers
74 views

How does the gradient increase the probabilities of the path with a positive reward in policy gradient?

Pieter Abbeel in his deep rl bootcamp policy gradient lecture derived the gradient of the utility function with respect to $\theta$ as $\nabla U(\theta) \approx \hat{g} = 1/m\sum_{i=1}^m \nabla_\...
1
vote
0answers
58 views

Is the reward following after time step $t+1$ collected based on current policy?

I am currently learning policy gradient methods from the Deep RL boot camp by Pieter Abbeel in which he explains the actor-critic algorithm derivation. At around minute 39, he explains that the sum ...
2
votes
1answer
104 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
3
votes
1answer
56 views

What does it mean to parameterise a policy in policy gradient methods?

Can you explain policy gradient methods and what it means for the policy to be parameterised? I am reading Sutton and Barto book on reinforcement learning and didn't understand well what it is, can ...
2
votes
0answers
30 views

PPO algorithm converges on only one action

I have taken some reference implementations of PPO algorithm and am trying to create an agent which can play space invaders . Unfortunately from the 2nd trial onwards (after training the actor and ...
1
vote
0answers
25 views

What if the rewards induced by an environment are related to the policy too?

Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
2
votes
0answers
42 views

What is the gradient of the Q function with respect to the policy's parameters?

I have been recently studying Actor-Critic algorithms, and I ran into the following question. Let $Q_{\omega}$ be the critic network, and $\pi_{\theta}$ be the actor. It is known that in order to ...
2
votes
1answer
58 views

How is the log-derivative trick of a trajectory derived?

I am looking at this formula which breaks down the gradient of $P(\tau |\theta)$ the first part is clear as is the derivative of $\log(x)$, but I do not see how the first formula is rearranged into ...
2
votes
1answer
53 views

How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?

I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
1
vote
0answers
35 views

Subtracting the entropy from our policy gradient will prevent our agent from being stuck in the local minimum?

In the information theory, the entropy is a measure of uncertainty in some system. Being applied to agent policy, entropy shows how much the agent is uncertain about which action to make. In math ...
1
vote
0answers
28 views

How can I design a DQN or policy gradient model to explore and collect all optimal solutions?

I am working to use DQN and Policy Gradient reinforcement learning models to solve classic maze escaping problems. So far, I have been able to train a model, which, after around 100 episodes, ...
3
votes
1answer
58 views

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

I'm working on a RL problem with the following properties: The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering ...