0
$\begingroup$

I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment.

I want to check for overestimation, therefore I'm plotting the action-state values for each episode. After 400+ episodes, I noticed that sometimes my predicted action-state values are negative. There are no negative rewards in this environment, only a +1 reward for each step taken.

Is this normal or is it somehow a bug?

Here you can see what happens on different runs: the run indicated by the brown line shows negative values for a few steps.Plot

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ If you’re using a NN then this would be normal, depending on how long you’ve trained for. As with any function approximators, there will always be some error, so unless you’re using an activation that forces the output to be non-negative, it is not out of the ordinary to see infeasible values (ie negative when all rewards are non-negative). It just means your model isn’t great $\endgroup$
    – David
    Dec 14, 2022 at 11:05

1 Answer 1

Reset to default
2
$\begingroup$

This is not really a bug but something undesirable, assuming you have a linear activation at the output layer, which can produce negative values.

If you want to prevent the your critic model from producing negative Q values, you could use a ReLU activation at the output layer.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .