How Can I Backpropagate My Network with PPO

Question

I am trying to implement PPO to my reinforcement agents. I have a classic neural network that represents the policy. I didn't quite understand how the PPO updates the network, according to what? There needs to be target values right? Or at least a loss function? For example in DQN a second network gives target Q values and the backpropagation is done according to that values for example with MSE loss function. But I don't know where I should put the returned value from PPO function. I don't get it. How can I backpropagate according to PPO values? Should I treat it like a target values of the network? If I should not, what are the target values or what is the loss?

Answer 1

I didn't quite understand how the PPO updates the network, according to what?

PPO implements what is called a policy gradient algorithm. What policy gradient is essentially doing is updating the policy in the direction of better return (return is the discounted sum of rewards during an episode). Let's say the policy is parametrized by neural network weights $\phi$. Then the weights update in the most basic policy gradient algorithm is computed as:

$$\phi = \phi + \alpha \nabla \log \pi_{\phi}(a_t,s_t) \cdot V_t,$$

where $V_t$ is the return. In case of PPO, $V_t$ is actually estimated (generalized) advantage function implemented by another neural network. This second neural network is often called critic, while your policy network is called an actor. Meaning that the actor is taking decisions, while the critic helps to train the actor by providing "an advice" in which direction to move.

There needs to be target values right? Or at least a loss function?

The loss function in PPO is

$$L^{PPO} = L^{PG} + c_1 L^{VF} - c_2 S,$$

where $L^{PG}$ is policy gradient loss calculated for actor performance, i.e. how good were actor's actions. In a simple policy gradient algorithm it would be $L^{PG} = -\sum_{t} \log \pi_{\phi}(a_t|s_t) \cdot V_t$, while in PPO it is a bit more sophisticated

$$ L^{PG}(\phi) = \sum_t \max \left( -r_t(\phi) \hat A_t, -\text{clip}(r_t(\phi), 1 - \epsilon, 1 + \epsilon) \hat A_t \right),$$

where $\hat A_t$ is empirical advantage function (the same as $V_t$ before) and $r_t(\phi) = \frac{\pi_{\phi}(a_t|s_t)}{\pi_{\phi_{\text{old}}}(a_t|s_t)}$ is the probability ratio of how better is the new policy is compared to the old one.

$L^{VF} = (V_{\theta}(s_t) - V_t^{\text{targ}})^2$ is value function loss function ($\theta$ are critic network parameters). This one is calculated for critic performance, i.e. how good it is able to estimate advantages. And $S$ is what is called "entropy" bonus (to increase the entropy, i.e. to stimulate exploration).

Should I treat it like a target values of the network?

No, not necessary. Note also that unlike DQN, PPO is an on-policy algorithm, which means that it throws away the episode information (states, rewards, actions) as soon as it is done with it.

For more details, see this excellent post about PPO details. There are also three Youtube videos explaining the subject. And I also wrote a post implementing PPO from scratch (first using REINFORCE to illustrate the simplest policy gradient algorithm). Also don't hesitate to read the original PPO paper.