2
$\begingroup$

I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target tracking. My question is how the new policy gets updated. My current implementation is as follows

  1. Make observation (distance from laser to target in pan and tilt)
  2. Pass observation to actor network which outputs a mean (std for now is fixed)
  3. I sample from a gaussian with the mean output from step 2
  4. Apply the command and observe the reward (1/L2 distance to target)
  5. collect N steps of experience, compute advantage and old log probabilities,
  6. train actor and critic

My question is this. I have my old log probabilities (probabilities of the actions taken given the means generated by the actor network), but I dont understand how the new probabilities are generated. At the onset of the very first minibatch my new policy is identical to my old policy as they are the same neural net. Given that in the model.fit function I am passing the same set of observations to generate 'y_pred' values, and I am passing in the actual actions taken as my 'y_true' values, the new policy should generate the exact same log probabilities as my old one. The only (slight) variation that makes the network update is from the entropy bonus, but my score np.exp(new_log_probs-old.log_probs) is nearly identically 1 because the policies are the same.

Should I be using a pair of networks similar to DDQN so there are some initial differences in the policies between the one used to generate the data and the one used for training?

$\endgroup$
  • $\begingroup$ Welcome to SE:AI! (Thanks for specifying conceptual, which is on-topic. However, if you don't get an answer here, might try CS or Overflow:) $\endgroup$ – DukeZhou Aug 26 at 23:50
1
$\begingroup$

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray too far away from the optimal point.

This means you repeat your step 6. epoch amount of times for the same batch of trajectories. Usually epoch is somewhere between 3 and 30 but it is a hyper-parameter you need to adjust. For the first repeat, the old and the new policy are the same, so their ratio should be 1. After the first update, the new probabilities will change due to the updated policy, whereas you will still need to use the old probabilities giving you a ratio different from 1. The old probabilities will stay the same during these epoch update steps, whereas your new probabilities will keep changing.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your response. When the ratio is 1, how does the new policy change so the ratio isn't one the second time around? Doesn't a ratio of one mean the gradient is zero? $\endgroup$ – Gideon Aug 27 at 16:09
  • 1
    $\begingroup$ @Gideon When the ratio is 1, the update reduces to a simple policy gradient update, for example to one used in the advantage actor-critic. The gradient is not zero, because your parameterized policy $\pi_\theta$ depends on $\theta$ and this gradient will exist. The ratio is just a multiplier telling you how much you can scale the gradient $\partial (\log{\pi_\theta})/\partial \theta$. $\endgroup$ – Hai Nguyen Aug 28 at 7:56
  • 1
    $\begingroup$ If you'd like to "cheat" a bit, you could peek into this repo (github.com/seungeunrho/minimalRL) that contains simple RL implementations in pytorch and compare for example the code for actor-critic and PPO. $\endgroup$ – Hai Nguyen Aug 28 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.