5 votes
Accepted

How to improve the reward signal when the rewards are sparse?

Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of ...
nbro's user avatar
  • 40.5k
3 votes
Accepted

What should I do when the potential value of a state is too high?

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward ...
Brenden Petersen's user avatar
3 votes

What should I do when the potential value of a state is too high?

I don't think the situation you're sketching should be a problem at all. If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the ...
Dennis Soemers's user avatar
  • 10.3k
2 votes
Accepted

Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)

Is the method itself defective or anything wrong with my code? There does indeed appear to be an issue with the code, the publications are fine (I know most of those authors and would very much trust ...
Dennis Soemers's user avatar
  • 10.3k
2 votes
Accepted

Why does potential-based reward shaping seem to alter the optimal policy in this case?

The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than ...
Dennis Soemers's user avatar
  • 10.3k

Only top scored, non community-wiki answers of a minimum length are eligible