I understand that this is the update for the parameters of a policy in REINFORCE:

$$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is usually the discounted future reward and $\pi_{\theta}\left(a_{t} \mid s_{t}\right)$ is the probability of taken the action that the agent took at time $t$. (Tell me if something is wrong here)

But I don't understand how that is passed into a neural network for backpropagation.

probs = policy.feedforward(state) returns the probabilities of taking each action, like [0.6, 0.4].

 action = choose_action_from(probs) will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

When it is time to update, we have

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?

If not, which loss function should I use in REINFORCE, and what are the labels?

If you need more explanation, I have this question on SO.

I understand that this is the update for the parameters of a policy in REINFORCE:

$$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is usually the discounted future reward and $\pi_{\theta}\left(a_{t} \mid s_{t}\right)$ is the probability of taken the action that the agent took at time $t$. (Tell me if something is wrong here)

However, I don't understand how to implement this with a neural network.

Let's say that probs = policy.feedforward(state) returns the probabilities of taking each action, like [0.6, 0.4].  action = choose_action_from(probs) will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

When it is time to update the parameters of the policy network, what should we do? Should we do something like the following?

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

And I only backpropagate this through one output neuron?

Which loss function should I use in this case? What would the labels be?

If you need more explanation, I have this question on SO.

I understand that this is the update for the parameters of a policy in REINFORCE:

$$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is usually the discounted future reward and $\pi_{\theta}\left(a_{t} \mid s_{t}\right)$ is the probability of taken the action that the agent took at time $t$. (Tell me if something is wrong here)

But I don’t understand how that is passed into a neural network for backpropagation.

I have this pseudocode

probs = policy.feedforward(state)

This returns the probabilities if taking each action, like [0.6, 0.4].

action = choose_action_from(probs) will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

Then later when it is time to update, is it:

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?

If you need more explanation, I have this question on SO.

I understand that this is the update for the parameters of a policy in REINFORCE:

$$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is usually the discounted future reward and $\pi_{\theta}\left(a_{t} \mid s_{t}\right)$ is the probability of taken the action that the agent took at time $t$. (Tell me if something is wrong here)

But I don't understand how that is passed into a neural network for backpropagation.

probs = policy.feedforward(state) returns the probabilities of taking each action, like [0.6, 0.4].

action = choose_action_from(probs) will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

When it is time to update, we have

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?

If not, which loss function should I use in REINFORCE, and what are the labels?

If you need more explanation, I have this question on SO.

I understand that this is the update for the parameters of a policy in REINFORCE:

$$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t} $$

Where 𝑣_𝑡 is usually the discounted future reward and 𝜋_𝜃(𝑎_𝑡|𝑠_𝑡) is the probability of taken the action that the agent took at time 𝑡. (Tell me if something is wrong here)

But I don’t understand how that is passed into a neural network for back propagation.
I have this pseudocode
probs = policy.feedforward(state)
This returns the probabilities if taking each actions, like: [0.6,0.4]

action = choose_action_from(probs) this will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

Then later when it is time to update, is it:

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?

If you need more explanation, I have this question on SO.

I understand that this is the update for the parameters of a policy in REINFORCE:

$$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is usually the discounted future reward and $\pi_{\theta}\left(a_{t} \mid s_{t}\right)$ is the probability of taken the action that the agent took at time $t$. (Tell me if something is wrong here)

But I don’t understand how that is passed into a neural network for backpropagation.

I have this pseudocode

probs = policy.feedforward(state)

This returns the probabilities if taking each action, like [0.6, 0.4].

action = choose_action_from(probs) will return the index of the probability chosen. For example, if it chose 0.6, the action would be 0.

Then later when it is time to update, is it:

gradient = policy.backpropagate(total_discounted_reward*log(probs[action])
policy.weights += gradient

Is this the right way to calculate the derivative of the loss and backpropagate it? And I only backpropagate this through one output neuron?

If you need more explanation, I have this question on SO.