# Generation of 'new log probabilities' in continuous action space PPO

I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target tracking. My question is how the new policy gets updated. My current implementation is as follows

1. Make observation (distance from laser to target in pan and tilt)
2. Pass observation to actor network which outputs a mean (std for now is fixed)
3. I sample from a gaussian with the mean output from step 2
4. Apply the command and observe the reward (1/L2 distance to target)
5. collect N steps of experience, compute advantage and old log probabilities,
6. train actor and critic

My question is this. I have my old log probabilities (probabilities of the actions taken given the means generated by the actor network), but I dont understand how the new probabilities are generated. At the onset of the very first minibatch my new policy is identical to my old policy as they are the same neural net. Given that in the model.fit function I am passing the same set of observations to generate 'y_pred' values, and I am passing in the actual actions taken as my 'y_true' values, the new policy should generate the exact same log probabilities as my old one. The only (slight) variation that makes the network update is from the entropy bonus, but my score np.exp(new_log_probs-old.log_probs) is nearly identically 1 because the policies are the same.

Should I be using a pair of networks similar to DDQN so there are some initial differences in the policies between the one used to generate the data and the one used for training?

• Welcome to SE:AI! (Thanks for specifying conceptual, which is on-topic. However, if you don't get an answer here, might try CS or Overflow:)
– DukeZhou
Aug 26, 2020 at 23:50

This means you repeat your step 6. epoch amount of times for the same batch of trajectories. Usually epoch is somewhere between 3 and 30 but it is a hyper-parameter you need to adjust. For the first repeat, the old and the new policy are the same, so their ratio should be 1. After the first update, the new probabilities will change due to the updated policy, whereas you will still need to use the old probabilities giving you a ratio different from 1. The old probabilities will stay the same during these epoch update steps, whereas your new probabilities will keep changing.
• @Gideon When the ratio is 1, the update reduces to a simple policy gradient update, for example to one used in the advantage actor-critic. The gradient is not zero, because your parameterized policy $\pi_\theta$ depends on $\theta$ and this gradient will exist. The ratio is just a multiplier telling you how much you can scale the gradient $\partial (\log{\pi_\theta})/\partial \theta$. Aug 28, 2020 at 7:56