I understand that this is the update for the parameters of a policy in REINFORCE:

$$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t} $$

Where 𝑣_𝑡 is usually the discounted future reward and 𝜋_𝜃(𝑎_𝑡|𝑠_𝑡) is the probability of taken the action that the agent took at time 𝑡. (Tell me if something is wrong here)

But I don’t understand how that is passed into a neural network for back propagation.
I have this pseudocode
probs = policy.feedforward(state)
This returns the probabilities if taking each actions, like: [0.6,0.4]

action = choose_action_from(probs) this will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

Then later when it is time to update, is it:

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?

If you need more explanation, I have this question on SO.