0
$\begingroup$

I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, especially PyTorch's implementation (found here). It is clear to me now that there are superior methods to train RL agents (PPO for instance), but as I read, these feel beyond my current intellectual or time resources. As such, I'd like to eek out as much power through modifications of REINFORCE as possible before determining how I might move on.

As such, are there modifications to the REINFORCE training algorithm that might yield benefits without straying far into new algorithm territory? Or perhaps, what is the "SOTA" version of REINFORCE?

For instance, perhaps a simple gradient clip in some way approximates some of PPO's benefits? Or maybe setting a baseline reward based on a rolling reward set of previous episodes?


If a specific context is useful, I'm applying this to a self-made simple grid environment where agents receive points namely for moving closer to and acquiring "targets." There are other rewards of lesser significance, but key is the environment is set for a sort of "continuous play" such that agents are very frequently receiving rewards (due to closeness) and occasionally receiving reward spikes (due to getting a target), but there is no true episode definition other than an arbitrary timestep length. I am not using batches (perhaps that is useful?), as I have found that gradually stepping up the episode length allows the agents quicker access to simpler rewards and appears a sort of scaffolding to more complex behavior. Agents might reasonably gather the "large" rewards in as few as 10-25 steps. Generally things are working fine, and I am most interested in how to extract as much value from the agent updating mechanism (REINFORCE) as possible.

$\endgroup$

1 Answer 1

4
$\begingroup$

One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the same folder in actor_critic.py.

In your implementation of REINFORCE, the gradient of the loss is calculated as: $$ L(\theta) = -\sum_{t=0}^T\hat{G}_t\log(\pi_\theta(a_t|s_t)) $$ with $$\hat{G}_t = \frac{\sum_{k=0}^t r_k - \hat{r}}{\hat{\sigma_r} + \epsilon}$$ where $\hat{r}$ is the empirical mean over the trajectory, $\hat{\sigma_r}$ the empirical standard deviation, and $\epsilon$ a constant to ensure the denominator is not too close to zero.

Now, actor_critic.py replaces $\hat{G}_t$ in the loss with $\hat{G}_t - V_\phi(s_t)$, called the advantage function, where $V_\phi$ is the state value function, which is a neural network with parameters $\phi$. $V_\phi$ is learnt, so the actual loss you optimize is: $$J(\theta,\phi) = L(\theta) + \sum_{t=0}^T\left(V_\phi(s_t) - \hat{G}_t\right)^2$$

This method is superior to your plain REINFORCE because the variance of the gradient updates is lower (which means learning is more steady and you obtain a better estimate of the optimal policy $\pi^*$).

$\endgroup$
2
  • $\begingroup$ Am I correct in understanding that, even though the actor and the critic are the same network except the last layer, that change is sufficient enough for the network to provide the state value? And event though the policy is providing the action at step t (which creates a new state t+1), the critic is providing the value for the new state (t+1)? $\endgroup$
    – Josh
    Sep 13, 2022 at 0:02
  • $\begingroup$ Yes, it's common for the value and the policy network to share most of their weights, except the final layer(s). You can play around with what's shared and what's not though, there are no rules. I'm not sure why you say that the critic is providing the value for the new state: the idea of the advantage is to tell you how much better the return you get by taking action $a_t$ is compared to the return of the "average" action (what the value function tells you) $\endgroup$ Sep 14, 2022 at 19:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .