In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to say, the real reward function is partially dependent on previously accumulated reward.
Is there any way to implement this kind of dynamic successfully?
I have tried to, but for some reason, the agent completely stops learning when I do this. I can implement a linear/additive reward function and the agent does learn good behaviors, but I feel that it's important for the agent to "understand" the true reward dynamic.
Essentially, here is the reward function I have:
reward = points_gained_this_step
But here what I need:
reward = points_gained_this_step*(total_score_so_far)
total_score_so_far = total_score_so_far + reward
Has anyone ever worked with something like this? Any ideas/insight for how to implement such a reward? I might be wrong, but it seems to me like part of the problem is exploding/vanishing gradients?
EDIT: I have already added "total_score_so_far" to my observation space.