2
$\begingroup$

I was going through the code by Andrej Karpathy on reinforcement learning using a policy gradient. I have some questions from the code.

  1. Where is the logarithm of the probability being calculated? Nowhere in the code I see him calculating that.

  2. Please explain to me the use of dlogps.append(y - aprob) line. I know this is calculating the loss, but how is this helping in a reinforcement learning environment, where we don't have the correct labels?

  3. How is policy_backward() working? How are the weights changing to the loss function mentioned above? More specifically, what's dh here?

|improve this question
$\endgroup$
1
+50
$\begingroup$
  1. logp seen in code is actually logit p which has this story behind:

Given a probability p, the corresponding odds are calculated as p / (1 – p). For example if p=0.75, the odds are 3 to 1: 0.75/0.25 = 3.

The logit function is simply the logarithm of the odds: logit(x) = log(x / (1 – x)).

Sigmoid near logp is like follows:

The inverse of the logit function is the sigmoid function. That is, if you have a probability p, sigmoid(logit(p)) = p.

Source: [1]

  1. In reinforcement learning we know in the end of game, if taken actions were successful or not. Then before next round we can adjust the gradients. From your link (commentary section):

For example in Pong we could wait until the end of the game, then take the reward we get (either +1 if we won or -1 if we lost), and enter that scalar as the gradient for the action we have taken (DOWN in this case). In the example below, going DOWN ended up to us losing the game (-1 reward). So if we fill in -1 for log probability of DOWN and do backprop we will find a gradient that discourages the network to take the DOWN action for that input in the future (and rightly so, since taking that action led to us losing the game).

  1. In the very same commentary section (later) there is a pic, and explanation of what h is. Unfortunately, you have to check it by yourself, pic was not compatible to be attached here. By describing the pic I could say h is equal to weights in hidden layer and in gradient case the dh is the derivative of h.

Roughly speaking the backpropagation is correcting the weights backwards the network after the round is done. More thorough explanation is in the mentioned comments section.

Sources:

[1] https://www.google.com/amp/s/nathanbrixius.wordpress.com/2016/06/04/functions-i-have-known-logit-and-sigmoid/amp/

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks a lot for this superb explanation. I do have one question. One other implementation uses one hot encoding to train the neural network but Andrej doesn't do that. Would you be able to be tell me why the difference? $\endgroup$ – Sarvagya Gupta Nov 20 at 15:49
  • $\begingroup$ Does that another neural network do the same thing (try to play pong)? I don't see how one hot decoding, or any decoding plays part on this task. For categorizing apples and oranges from peaches I see it useful. $\endgroup$ – mico Nov 20 at 17:50
  • $\begingroup$ See: hackernoon.com/… $\endgroup$ – mico Nov 20 at 17:59
  • $\begingroup$ So this is Keras implementation of Karpathy's code. This person is also implementing RL on pong. Checkout line 108. I think the person is doing one-hot encoding here github.com/Alexander-H-Liu/… $\endgroup$ – Sarvagya Gupta Nov 21 at 4:35
  • $\begingroup$ Maybe ask another question about the comparison of these solutions? $\endgroup$ – mico Nov 22 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.