4
$\begingroup$

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty when a wrong move is made. I'm using a neural network with stochastic gradient decent to learn the policy. To do this, my loss is essentially the cross-entropy loss of the action distribution multiplied by the discounted rewards, where most often the rewards are positive.

But how do I handle negative rewards? Since the loss will occasionally go negative, it will think these actions are very good, and will strengthen the weights in the direction of the penalties. Is this correct, and if so, what can I do about it?

Edit: In thinking about this a little more, SGD doesn't necessarily directly weaken weights, it only strengthens weights in the direction of the gradient and as a side-effect, weights get diminished for other states outside the gradient, correct? So I can simply set reward=0 when the reward is negative, and those states will be ignored in the gradient update. It still seems unproductive to not account for states that are really bad, and it'd be nice to include them somehow. Unless I'm misunderstanding something fundamental here.

$\endgroup$
  • $\begingroup$ Just clarifying, this is for discrete actions? $\endgroup$ – Jaden Travnik Oct 17 '17 at 4:01
3
$\begingroup$

It depends on your loss function, but you probably need to tweak it.

If you are using an update rule like loss = -log(probabilities) * reward, then your loss is high when you unexpectedly got a large reward—the policy will update to make that action more likely to realize that gain.

Conversely, if you get a negative reward with high probability, this will result in negative loss—however, in minimizing this loss, the optimizer will attempt to make this loss "even more negative" by making the log probability more negative (i.e. by making the probability of that action less likely)—so it kind of does what we want.

However, now improbable large negative losses are punished more than the more than likely ones, when we probably want the opposite. Hence, loss = -log(1-probabilities) * reward might be more appropriate when the reward is negative.

$\endgroup$
  • $\begingroup$ While negative rewards being valid, wouldn't they really screw up your gradient - in the sense that the gradient for positive and negative rewards would be highly asymmetric? Positive rewards will cause a diminishing gradient the closer the action probability goes to 1, whereas negative rewards will cause a strongly increasing gradient the closer the action probability goes to 0. $\endgroup$ – user12889 Jul 5 '18 at 0:33
  • $\begingroup$ I think you meant loss = -log(probabilities) * reward (minus before the log) $\endgroup$ – Zsolt Safrany Feb 13 at 4:47
  • $\begingroup$ @user12889 I'm thinking it is symmetric in the sense that a -1 reward has precisely the opposite gradient of a +1 reward. I think the asymmetry you're describing is related to using negative log probabilities as opposed to using something like 1-p. But this asymmetry of the loss function will impact both positive and negative rewards the same. $\endgroup$ – Tahlor Feb 13 at 21:00
  • $\begingroup$ @Tahlor Yes, but over time, as the model learns, you would expect probabilities for +1 rewards to move closer to 1 and probabilities for -1 rewards to move closer to 0, which then leads to generally highly non-symmetric gradients for +1 and -1 rewards. $\endgroup$ – user12889 Feb 14 at 22:27
  • $\begingroup$ Oh right, so couldn't you just invert and shift your loss function for negative rewards? So -log(1-probabilities) * reward for negative rewards and the regular one for positive rewards to preserve that symmetry? $\endgroup$ – Tahlor Feb 15 at 1:17
2
$\begingroup$

Cross-entropy loss will always be positive, because the probability is 0~1, -ln(p) will always be positive.

$\endgroup$
  • 2
    $\begingroup$ Right, I think the issue is he's multiplying -ln(p) by a potentially negative number (his reward). This loss function expects only positive rewards. $\endgroup$ – Tahlor Apr 15 '18 at 3:32
  • $\begingroup$ @Tahlor I think you are right about the reward needing to be positive. I'm having the same problem and came to the same conclusion. However I'm having a hard time finding an authoritative source for this. Do you have one at hand? $\endgroup$ – user12889 Jul 5 '18 at 0:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.