For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

46 questions
Filter by
Sorted by
Tagged with
64 views

### What is the difference between Sutton's and Levine's REINFORCE algorithm?

I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below. In particular, what does $i$ represent in the REINFORCE algorithm? ...
30 views

### Reinforcement Learning on quantum circuit

I am trying to teach an agent to make any random 1-qubit state reach uniform superposition. So basically, the full circuit will be ...
26 views

### Should I consider mean or sampled value for action selection in ppo algorithm?

When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
14 views

### Deciding std. deviation for policy network output?

When I try to fit a Normal Distribution to the output of a policy network, for a continuous action space problem, what should be its standard deviation? mean for the distribution will directly be the ...
71 views

### Eligibility vector for softmax policy with policy gradients

There is this nice result for policy gradients that the gradient of some performance measure, $\nabla v_{\pi_{\theta}}(s_0)$ (here, in the episodic case for the starting state $s_0$ and policy $\pi$, ...
44 views

### Why is the stationary distribution independent of the initial state in the proof of the policy gradient theorem?

I was going through the proof of the policy gradient theorem here: https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#svpg In the section "Proof of Policy Gradient ...
43 views

### What is the effect of picking action deterministicly at inference with Policy Gradient Methods?

In policy gradient methods such as A3C/PPO, the output from the network is probabilities for each of the actions. At training time, the action to take is sampled from the probability distribution. ...
24 views

### How to set the multiple continuous actions with constraints

I want to build a Deep Reinforcement Learning Model for Asset allocation. Background: I have 7 stock indexes from different markets, and I want to build a policy to produce the action (likes whether ...
41 views

### Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
58 views

29 views

### Neural network with logical hidden layer - how to train it? Is it policy gradient problem? Chaining NNs?

I am doing neural machine translation task from language S to language T via interlingua L. So - there is the structure: ...
2k views

### Why does is make sense to normalize rewards per episode in reinforcement learning?

In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so rewards = (rewards - rewards.mean()) / (rewards.std() + eps) on ...
30 views

### How to include exploration in Gaussian policy

When dealing with continuous action spaces, a common choice when designing a policy in policy gradient methods is to learn mean and variance of actions for a specific state and then simply sample from ...
28 views

### Impact of Varying Length Trajectories on Policy Gradient Optimization

As the question states, I am wondering how, if at all, a varying length of a trajectory (series of state,action pairs) will impact training/performance of policy gradient algorithms such as PPO, TRPO ...
210 views

### Can gradient descent training be used for nonsmooth loss functions?

I have non-smooth loss function - e.g. loss(x)=min(x, 0.5). Can gradient descent be used for training neural networks with such functions. Can gradient descent be used for fairly general, ...
393 views

### Why does the “reward to go” trick in policy gradient methods work?

In policy gradient method, there's a trick to reduce a variance of policy gradient. We use causality, and remove part of the sum over rewards so that only actions happened after the reward are taken ...
5k views

### What is the relation between Q-learning and policy gradients methods?

As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain ...