2
$\begingroup$

In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to say, the real reward function is partially dependent on previously accumulated reward.

Is there any way to implement this kind of dynamic successfully?

I have tried to, but for some reason, the agent completely stops learning when I do this. I can implement a linear/additive reward function and the agent does learn good behaviors, but I feel that it's important for the agent to "understand" the true reward dynamic.

Essentially, here is the reward function I have:

reward = points_gained_this_step

But here what I need:

reward = points_gained_this_step*(total_score_so_far)
total_score_so_far = total_score_so_far + reward

Has anyone ever worked with something like this? Any ideas/insight for how to implement such a reward? I might be wrong, but it seems to me like part of the problem is exploding/vanishing gradients?

EDIT: I have already added "total_score_so_far" to my observation space.

$\endgroup$

1 Answer 1

3
$\begingroup$

The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent is going to need to know this.

You may still have some problems after doing this. The final return that the agent needs to predict is likely to have the following traits:

  • A value that could range by orders of magnitude, making it hard to scale loss functions. You may be able to base the loss function for value predictions on mean relative error to help with this.

  • Large variance, making it difficult to learn expected returns, especially from the impact of early decisions. If your immediate rewards have any random element, this could be a major problem.

If value-based methods such as DQN are struggling in your case, and the optimal policy function is straightfoward, then you may want to try policy gradient methods, maybe even basic REINFORCE, to avoid needing to predict returns whilst dealing with such a high variance. You will still need to take care with scaling, as policy gradient methods scale update steps based on the return.

$\endgroup$
2
  • $\begingroup$ Thank you very much for your response. I forgot to mention, I have actually already added the accumulated reward to my observation space. While I'm not sure, I doubt the optimal policy is straightforward (it's complicated enough that I myself was unable to learn the optimal policy by manually playing with the data). I agree with every point you make, particularly the two bullet points. I had an idea to make my episodes shorter (less timesteps per episode), so that 1) there isn't enough time for rewards to get huge/tiny and 2) the "impact of early decisions" is reduced. What do you think? $\endgroup$ May 31, 2022 at 17:00
  • 1
    $\begingroup$ @VladimirBelik That's a good idea IMO. If you can simplify the environment and find an approach that works there, you may be able to scale it back up to the full environment by degrees based on what you discover. I cannot say what is likely to work though, as I have not really studied an environment with reward behaviour that you describe in the question $\endgroup$ May 31, 2022 at 17:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .