For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

150 questions
Filter by
Sorted by
Tagged with
13 views

### What specifically is the gradient of the log of the probability in policy gradient methods (reinforcement learning)

I am getting tripped up slightly by how specifically the gradient is calculated in policy gradient methods (just the intuitive understanding of it). https://math.stackexchange.com/questions/2845971/...
21 views

### FrozenLake-v0 not training using REINFORCE

I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all. I have used the same neural architecture ...
15 views

### How to fix loss?

I am trying to use the Vanilla Policy Gradient algorithm to solve MountainCar-v0. I have taken the code from pytorch repo. ...
65 views

### Why exactly was previously believed that the deterministic policy gradient did not exist?

I'm reading the paper Deterministic Policy Gradient Algorithms, David Silver et al. First of all, in the introduction, the author says that It was previously believed that the deterministic policy ...
33 views

### Why does this PPO agent doesn't learn at all?

I am trying to code a Proximal Policy Optimization algorithm in Pytorch by myself for the OpenAI gym preduum-v0 environment. However, I find my learning curve is ...
21 views

### Can we learn a policy network via a sequence of manually determined actions?

In policy gradients, is it possible to learn the policy if the chain of actions is selected and performed manually/externally (e.g. by myself or by someone else who I have no influence over)? For ...
44 views

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o_t$) with a camera at each location (action $... 0answers 18 views ### multi agent deep deterministic policy gradient for discrete actions I am solving a multi agent problem where each agent has a critic and actor. The problem I am solving has discrete actions and discrete states. I came cross multi-agent deep deterministic policy ... 0answers 21 views ### DDPG or PPO don't work well with my custom non gym environment I have a project to control a robot with right and left wheel speeds, and my step time is not constant. Because my outputs are continuous (right wheel speed, left wheel speed, and time step), I try to ... 1answer 36 views ### How to interpret the policy gradient expression in reinforcement learning? I'm currently going through the OpenAI's spinning up introduction course to reinforcement learning. On one of the final sections, they derive an expression for the gradient of the undiscounted return ... 1answer 36 views ### Why adding a baseline doesn't affect the policy gradient? On the OpenAI's Spinning Up, they justify the fact that adding a baseline$b(s_t)$in the policy gradient doesn't change its gradient by saying that this is an immediate consequence of the EGLP Lemma ... 0answers 19 views ### Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient. 1, in my question, each agent has its own action space. When doing the sampling, for each ... 1answer 72 views ### How does the neural network learn when used in the REINFORCE algorithm? As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just 1 loss value. How does the neural network learn to differentiate between good and bad ... 0answers 16 views ### A2C value function outputs keep increasing I was implementing the A2C algorithm with as close to baseline setup as possible, and this is the code I came up with. The problem is that even after multiple rechecks, the program isn't showing ... 0answers 12 views ### How do I quantify the difference in sample efficiency for two almost similar methods? I am comparing my coded TD3 (Twin-Delayed DDPG) and the same TD3 (same hyperparameters) but with Priority Replay Buffer instead of a normal Replay Buffer. From what I have read, PER (Priority ... 1answer 44 views ### How to formulate discounted return in cartpole? I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return: finite horizon undiscounted ... 1answer 483 views ### Are policy gradient methods good for large discrete action spaces? I have seen this question asked primarily in the context of continuous action spaces. I have a large action space (~2-4k discrete actions) for my custom environment that I cannot reduce down further: ... 1answer 93 views ### Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions i'm trying to solve a problem in which i need to carry out reinforcement learning with multiple simultaneous actions in continuous action space . i checked the multiagent structure; however, im trying ... 1answer 90 views ### Understanding the On-policy state distribution for episodic tasks with$\gamma \in (0,1)$In Sutton and Barto's Reinforcement Learning: An Introduction, section 9.2 (page 199) (here is a screenshot) describes the on-policy distribution in episodic tasks, with$\gamma =1$, as being \begin{... 0answers 216 views ### PPO in continuous control not working I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <... 1answer 51 views ### Is the policy gradient expression in Fundamentals of Deep Learning wrong? I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning. Here is the whole paragraph: Policy Learning via Policy ... 0answers 22 views ### Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one? I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL ... 1answer 104 views ### Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss? I read the book "Foundation of Deep Reinforcement Learning, Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function: $$J\left(\... 0answers 21 views ### Understanding advantage estimator in proximal policy optimization I was reading Proximal Policy Optimization paper. It states following: The advantage estimator used is: \hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)... 0answers 26 views ### Which off-policy policy gradient estimator has lower variance? Let \pi_\theta be a target policy and \beta_\theta be a behavior policy. I have seen the following 2 policy gradient estimators in the literature:$$ \operatorname*{E}_{\tau \sim \beta_\theta} \... 1answer 23 views ### How to have zero value or a value between 200 and 400 in the output of a deep learning model? I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have ... 1answer 55 views ### Why is the behaviour policy denoted by$\beta$and the exploration policy by$ \mu'$in the DDPG paper? I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy. Lillicrap et al. denote the policy gradient by $$\... 1answer 93 views ### It is mathematically correct to use a Policy Gradient method for 1-step trajectories? I have come across a Google paper that uses the REINFORCE algorithm (a Policy Gradient Method) for a case where the trajectory of the episodes it proposes would be ... 0answers 57 views ### Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285) I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of ... 0answers 38 views ### Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ... 0answers 38 views ### Understanding policy gradient derivation with importance sampling I was watching a lecture on Policy Gradient and had difficulty following it when importance sampling was introduced. It was shown that the gradient of the objective can be written as$$\nabla_\theta U(... 0answers 70 views ###$\nabla \log \pi\$ with respect to some parameters constantly being zero

I am new to reinforcement learning. May I ask a simple (and maybe a bit silly) question here? I am trying to use the "one-step actor-critic" method to train a robot on a gridworld. Let's ...
25 views

### Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?

I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard. My ...
220 views

### Designing Policy-Network for Deep-RL with Large, Variable Action Space

I am attempting a project involving training an agent to play a game using deep reinforcement learning. This project has a few features that complicate the design of the neural network: The action ...
97 views

50 views

### Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: \begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} ...
47 views

### What are the disadvantages of actor-only methods with respect to value-based ones?

While the advantages of actor-only algorithms, the ones that search directly the policy without the use of the value function, are clear (possibility of having a continuous action space, a stochastic ...
656 views

### What is the loss for policy gradients with continuous actions?

I know with policy gradients used in an environment with a discrete action space are updated with $$\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}$$ ...
266 views

### What, exactly, does the REINFORCE update equation mean?

I understand that this is the update for the parameters of a policy in REINFORCE: $$\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}$$ Where 𝑣𝑡 is ...
80 views

### Why does (not) the distribution of states depend on the policy parameters that induce it?

I came across the following proof of what's commonly referred to as the log-derivative trick in policy-gradient algorithms, and I have a question - While transitioning from the first line to the ...
34 views

### What's an example of a simple policy but a complex value function?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
154 views

### Generation of 'new log probabilities' in continuous action space PPO

I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target ...
1k views

### How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?

In section 3 of the paper Continuous control with deep reinforcement learning, the authors write As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & ...
131 views

### Why does REINFORCE work at all?

Here's a screenshot of the popular policy-gradient algorithm from Sutton and Barto's book - I understand the mathematical derivation of the update rule - but I'm not able to build intuition as to why ...
30 views

### Customized food for persons based on their profile using Reinforcement learning

I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based ...