logp
seen in code is actually logit p
which has this story behind:
Given a probability p, the corresponding odds are calculated as p / (1 – p). For example if p=0.75, the odds are 3 to 1: 0.75/0.25 = 3.
The logit function is simply the logarithm of the odds: logit(x) = log(x / (1 – x)).
Sigmoid near logp
is like follows:
The inverse of the logit function is the sigmoid function. That is, if you have a probability p, sigmoid(logit(p)) = p.
Source: [1]
- In reinforcement learning we know in the end of game, if taken actions were successful or not. Then before next round we can adjust the gradients. From your link (commentary section):
For example in Pong we could wait until the end of the game, then take the reward we get (either +1 if we won or -1 if we lost), and enter that scalar as the gradient for the action we have taken (DOWN in this case). In the example below, going DOWN ended up to us losing the game (-1 reward). So if we fill in -1 for log probability of DOWN and do backprop we will find a gradient that discourages the network to take the DOWN action for that input in the future (and rightly so, since taking that action led to us losing the game).
- In the very same commentary section (later) there is a pic, and explanation of what h is. Unfortunately, you have to check it by yourself, pic was not compatible to be attached here. By describing the pic I could say h is equal to weights in hidden layer and in gradient case the dh is the derivative of h.
Roughly speaking the backpropagation is correcting the weights backwards the network after the round is done. More thorough explanation is in the mentioned comments section.
Sources:
[1] https://www.google.com/amp/s/nathanbrixius.wordpress.com/2016/06/04/functions-i-have-known-logit-and-sigmoid/amp/