10
$\begingroup$

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural network with stochastic gradient descent to learn the policy. To do this, my loss is essentially the cross-entropy loss of the action distribution multiplied by the discounted rewards, where most often the rewards are positive.

But how do I handle negative rewards? Since the loss will occasionally go negative, it will think these actions are very good, and will strengthen the weights in the direction of the penalties. Is this correct, and if so, what can I do about it?


Edit:

In thinking about this a little more, SGD doesn't necessarily directly weaken weights, it only strengthens weights in the direction of the gradient and as a side-effect, weights get diminished for other states outside the gradient, correct? So I can simply set reward=0 when the reward is negative, and those states will be ignored in the gradient update. It still seems unproductive to not account for states that are really bad, and it'd be nice to include them somehow. Unless I'm misunderstanding something fundamental here.

$\endgroup$
2
  • $\begingroup$ Just clarifying, this is for discrete actions? $\endgroup$ Oct 17 '17 at 4:01
  • $\begingroup$ Did you ever get to the bottom of this? I'm still struggling with it. The issue for me seems to be that with a negative reward, the loss can be driven to -inf in the limit. I think the optimizer biases strongly toward driving the negative reward cases to zero, rather than pushing the positive reward cases to 1. $\endgroup$
    – Mastiff
    Jul 27 at 16:06
6
$\begingroup$

It depends on your loss function, but you probably need to tweak it.

If you are using an update rule like loss = -log(probabilities) * reward, then your loss is high when you unexpectedly got a large reward—the policy will update to make that action more likely to realize that gain.

Conversely, if you get a negative reward with high probability, this will result in negative loss—however, in minimizing this loss, the optimizer will attempt to make this loss "even more negative" by making the log probability more negative (i.e. by making the probability of that action less likely)—so it kind of does what we want.

However, now improbable large negative losses are punished more than the more than likely ones, when we probably want the opposite. Hence, loss = -log(1-probabilities) * reward might be more appropriate when the reward is negative.

$\endgroup$
5
  • $\begingroup$ While negative rewards being valid, wouldn't they really screw up your gradient - in the sense that the gradient for positive and negative rewards would be highly asymmetric? Positive rewards will cause a diminishing gradient the closer the action probability goes to 1, whereas negative rewards will cause a strongly increasing gradient the closer the action probability goes to 0. $\endgroup$
    – user12889
    Jul 5 '18 at 0:33
  • $\begingroup$ I think you meant loss = -log(probabilities) * reward (minus before the log) $\endgroup$ Feb 13 '19 at 4:47
  • $\begingroup$ @user12889 I'm thinking it is symmetric in the sense that a -1 reward has precisely the opposite gradient of a +1 reward. I think the asymmetry you're describing is related to using negative log probabilities as opposed to using something like 1-p. But this asymmetry of the loss function will impact both positive and negative rewards the same. $\endgroup$
    – Tahlor
    Feb 13 '19 at 21:00
  • $\begingroup$ @Tahlor Yes, but over time, as the model learns, you would expect probabilities for +1 rewards to move closer to 1 and probabilities for -1 rewards to move closer to 0, which then leads to generally highly non-symmetric gradients for +1 and -1 rewards. $\endgroup$
    – user12889
    Feb 14 '19 at 22:27
  • $\begingroup$ Oh right, so couldn't you just invert and shift your loss function for negative rewards? So -log(1-probabilities) * reward for negative rewards and the regular one for positive rewards to preserve that symmetry? $\endgroup$
    – Tahlor
    Feb 15 '19 at 1:17
2
$\begingroup$

The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.

$\endgroup$
2
  • 2
    $\begingroup$ Right, I think the issue is he's multiplying -ln(p) by a potentially negative number (his reward). This loss function expects only positive rewards. $\endgroup$
    – Tahlor
    Apr 15 '18 at 3:32
  • 2
    $\begingroup$ @Tahlor I think you are right about the reward needing to be positive. I'm having the same problem and came to the same conclusion. However I'm having a hard time finding an authoritative source for this. Do you have one at hand? $\endgroup$
    – user12889
    Jul 5 '18 at 0:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.