# Which loss function should I use in REINFORCE, and what are the labels?

I understand that this is the update for the parameters of a policy in REINFORCE:

$$\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t},$$ where $$v_t$$ is usually the discounted future reward and $$\pi_{\theta}\left(a_{t} \mid s_{t}\right)$$ is the probability of taken the action that the agent took at time $$t$$. (Tell me if something is wrong here)

However, I don't understand how to implement this with a neural network.

Let's say that probs = policy.feedforward(state) returns the probabilities of taking each action, like [0.6, 0.4]. action = choose_action_from(probs) will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

When it is time to update the parameters of the policy network, what should we do? Should we do something like the following?

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])


And I only backpropagate this through one output neuron?

Which loss function should I use in this case? What would the labels be?

If you need more explanation, I have this question on SO.

• I use a cross entropy loss when I implement this in PyTorch. I tried doing a forward pass of the network evaluated at the current action and taking the log but it didn't seem to work when I implemented it, so I stuck with cross entropy loss. Sep 16, 2020 at 16:31
• Is this question related to a specific library (eg. pytorch or tensorflow)? Or is this just a question for a more specific but still general pseudocode? Sep 16, 2020 at 16:42
• @David Ireland Cross entropy with what? What would be the label, y hat, or correct answer in the equation? Sep 16, 2020 at 16:51
• @Hai Nguyen It is a lot about the general pseudocode, but if you click on the link you can see I built my own neural network out of just NumPy. Sep 16, 2020 at 16:53
• @S2673 It is cross-entropy loss using as the label the action you actually took during sampling. Multiplying by the value function is really critical addition though, it can even reverse the sign of all the gradients (which makes sense - if you have chosen an action and it was a bad choice, you want to reduce the chance of taking it again) Sep 16, 2020 at 17:05

• @S2673 The algorithm doesn't change in this situation. Say your NN outputs the mean parameter of the Gaussian, then $\log_\pi(a_t | s_t)$ is just the log of the normal density evaluated at the action you took where the mean parameter in the density is the output of your NN. You are then able to backpropagate through this to update the weights of your network. Jan 7, 2021 at 11:22