For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

194 questions
Filter by
Sorted by
Tagged with
52 views

### In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?

Taking as an example the Advantage Actor Critic, the objective function is: \begin{equation} \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\... 18 views

### Can A2C deal with a reward that is decided later than action selection?

I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ...
51 views

### Discount factors in REINFORCE Algorithm: Difference in two definitions

I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/): Here, at every step in an ...
51 views

### Why policy gradient theorem has two different forms?

I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
24 views

### Policy Gradient in Partial Observability

Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
106 views

### Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
29 views

50 views

### Can objective function and gradient be unlimited in reinforcement learning?

I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
301 views

### How do we estimate the value of a stochastic policy?

I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
1 vote
62 views

### Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)

I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next ...
53 views

### Does the policy search work if there is no state to state dependency through actions?

There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
1 vote
237 views

### Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
1 vote
375 views

### Does importance sampling really improve sampling efficiency of TRPO or PPO?

Vanilla policy gradient has a loss function: $$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$ while in TRPO it is: \mathcal{L}_{\pi_{\...
366 views

### Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
1 vote
324 views

### Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

Consider the following algorithm from the textbook titled Reinforcement Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart While playing the game for the generation of ...