10
$\begingroup$

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural network with stochastic gradient descent to learn the policy. To do this, my loss is essentially the cross-entropy loss of the action distribution multiplied by the discounted rewards, where most often the rewards are positive.

But how do I handle negative rewards? Since the loss will occasionally go negative, it will think these actions are very good, and will strengthen the weights in the direction of the penalties. Is this correct, and if so, what can I do about it?

Edit:

In thinking about this a little more, SGD doesn't necessarily directly weaken weights, it only strengthens weights in the direction of the gradient and as a side-effect, weights get diminished for other states outside the gradient, correct? So I can simply set reward=0 when the reward is negative, and those states will be ignored in the gradient update. It still seems unproductive to not account for states that are really bad, and it'd be nice to include them somehow. Unless I'm misunderstanding something fundamental here.

Improve this question
$\endgroup$
2
  • $\begingroup$ Just clarifying, this is for discrete actions? $\endgroup$
    – Jaden Travnik
    Commented Oct 17, 2017 at 4:01
  • $\begingroup$ Did you ever get to the bottom of this? I'm still struggling with it. The issue for me seems to be that with a negative reward, the loss can be driven to -inf in the limit. I think the optimizer biases strongly toward driving the negative reward cases to zero, rather than pushing the positive reward cases to 1. $\endgroup$
    – Mastiff
    Commented Jul 27, 2021 at 16:06

2 Answers 2

Reset to default
7
$\begingroup$

It depends on your loss function, but you probably need to tweak it.

If you are using an update rule like loss = -log(probabilities) * reward, then your loss is high when you unexpectedly got a large reward—the policy will update to make that action more likely to realize that gain.

Conversely, if you get a negative reward with high probability, this will result in negative loss—however, in minimizing this loss, the optimizer will attempt to make this loss "even more negative" by making the log probability more negative (i.e. by making the probability of that action less likely)—so it kind of does what we want.

However, now improbable large negative losses are punished more than the more than likely ones, when we probably want the opposite. Hence, loss = -log(1-probabilities) * reward might be more appropriate when the reward is negative.

Improve this answer
$\endgroup$
5
  • $\begingroup$ While negative rewards being valid, wouldn't they really screw up your gradient - in the sense that the gradient for positive and negative rewards would be highly asymmetric? Positive rewards will cause a diminishing gradient the closer the action probability goes to 1, whereas negative rewards will cause a strongly increasing gradient the closer the action probability goes to 0. $\endgroup$
    – user12889
    Commented Jul 5, 2018 at 0:33
  • $\begingroup$ I think you meant loss = -log(probabilities) * reward (minus before the log) $\endgroup$
    – Zsolt Safrany
    Commented Feb 13, 2019 at 4:47
  • $\begingroup$ @user12889 I'm thinking it is symmetric in the sense that a -1 reward has precisely the opposite gradient of a +1 reward. I think the asymmetry you're describing is related to using negative log probabilities as opposed to using something like 1-p. But this asymmetry of the loss function will impact both positive and negative rewards the same. $\endgroup$
    – Tahlor
    Commented Feb 13, 2019 at 21:00
  • $\begingroup$ @Tahlor Yes, but over time, as the model learns, you would expect probabilities for +1 rewards to move closer to 1 and probabilities for -1 rewards to move closer to 0, which then leads to generally highly non-symmetric gradients for +1 and -1 rewards. $\endgroup$
    – user12889
    Commented Feb 14, 2019 at 22:27
  • $\begingroup$ Oh right, so couldn't you just invert and shift your loss function for negative rewards? So -log(1-probabilities) * reward for negative rewards and the regular one for positive rewards to preserve that symmetry? $\endgroup$
    – Tahlor
    Commented Feb 15, 2019 at 1:17
3
$\begingroup$

The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Right, I think the issue is he's multiplying -ln(p) by a potentially negative number (his reward). This loss function expects only positive rewards. $\endgroup$
    – Tahlor
    Commented Apr 15, 2018 at 3:32
  • 2
    $\begingroup$ @Tahlor I think you are right about the reward needing to be positive. I'm having the same problem and came to the same conclusion. However I'm having a hard time finding an authoritative source for this. Do you have one at hand? $\endgroup$
    – user12889
    Commented Jul 5, 2018 at 0:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .