# What, exactly, does the REINFORCE update equation mean?

I understand that this is the update for the parameters of a policy in REINFORCE:

$$\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}$$

Where 𝑣𝑡 is usually the discounted future reward and 𝜋𝜃(𝑎𝑡|𝑠𝑡) is the probability of taken the action that the agent took at time 𝑡. (Tell me if something is wrong here)

But I don’t understand how that is passed into a neural network for back propagation.
I have this pseudocode
probs = policy.feedforward(state)
This returns the probabilities if taking each actions, like: [0.6,0.4]

action = choose_action_from(probs) this will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

Then later when it is time to update, is it:

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient


Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?

If you need more explanation, I have this question on SO.

• I use a cross entropy loss when I implement this in PyTorch. I tried doing a forward pass of the network evaluated at the current action and taking the log but it didn't seem to work when I implemented it, so I stuck with cross entropy loss. – David Ireland Sep 16 '20 at 16:31
• Is this question related to a specific library (eg. pytorch or tensorflow)? Or is this just a question for a more specific but still general pseudocode? – Hai Nguyen Sep 16 '20 at 16:42
• @David Ireland Cross entropy with what? What would be the label, y hat, or correct answer in the equation? – S2673 Sep 16 '20 at 16:51
• @Hai Nguyen It is a lot about the general pseudocode, but if you click on the link you can see I built my own neural network out of just NumPy. – S2673 Sep 16 '20 at 16:53
• @S2673 It is cross-entropy loss using as the label the action you actually took during sampling. Multiplying by the value function is really critical addition though, it can even reverse the sign of all the gradients (which makes sense - if you have chosen an action and it was a bad choice, you want to reduce the chance of taking it again) – Neil Slater Sep 16 '20 at 17:05

## 1 Answer

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.

• Okay, thank you. It was not working with my code so I had to build a new, simpler one to test it and it worked. That’s why it took so long. There’s just one more thing... How do I know to do this given the equation in my question? Or is this not what the equation meant? – S2673 Sep 18 '20 at 22:06
• Should have clarified in my answer that I was referring to the cross entropy loss in pytorch — if you read the documentation you see that this loss function is the log of the policy for the action you give it. – David Ireland Sep 19 '20 at 23:07
• Thanks. Do you, by any chance, know how this works for a policy gradient with a continuous action space where the neural network outputs a Gaussian distribution? The equation for this update is the same but it is the log of a different output. – S2673 Sep 25 '20 at 1:20
• @S2673 The algorithm doesn't change in this situation. Say your NN outputs the mean parameter of the Gaussian, then $\log_\pi(a_t | s_t)$ is just the log of the normal density evaluated at the action you took where the mean parameter in the density is the output of your NN. You are then able to backpropagate through this to update the weights of your network. – David Ireland Jan 7 at 11:22
• Thank you. I had figured this out a bit ago. – S2673 Jan 8 at 3:24