Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward function, and how should you avoid them? What information from your problem should you take into consideration when going about it?
Let us presume that our environment is fully observable MDP.
Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any other situation).
However, in certain cases, the specification of the reward function can be a difficult task [1, 2, 3] because there are many (often unknown) factors that could affect the performance of the RL agent. For example, consider the driving task, i.e. you want to teach an agent to drive e.g. a car. In this scenario, there are so many factors that affect the behavior of a driver. How can we incorporate and combine these factors in a reward function? How do we deal with unknown factors?
So, often, designing a reward function is a trial-and-error and engineering process (so there is no magic formula that tells you how to design a reward function in all cases). More precisely, you define an initial reward function based on your knowledge of the problem, you observe how the agent performs, then tweak the reward function to achieve greater performance (for example, in terms of observable behavior, so not in terms of the collected reward; otherwise, this would be an easy problem: you could just design a reward function that gives infinite reward to the agent in all situations!). For example, if you have trained an RL agent to play chess, maybe you observed that the agent took a lot of time to converge (i.e. find the best policy to play the game), so you could design a new reward function that penalizes the agent for every non-win move (maybe it will hurry up!).
Of course, this trial-and-error approach is not ideal, and it can sometimes be impractical (because maybe it takes a lot of time to train the agent) and lead to misspecified reward signals.
It is well known that the misspecification of the reward function can have unintended and even dangerous consequences [5]. To overcome the misspecification of rewards or improve the reward functions, you have some options, such as
Learning from demonstrations (aka apprenticeship learning), i.e. do not specify the reward function directly, but let the RL agent imitate another agent's behavior, either to
Incorporate human feedback [9] in the RL algorithms (in an interactive manner)
Transfer the information in the policy learned in another but similar environment to your environment (i.e. use some kind of transfer learning for RL [10])
Of course, these solutions or approaches can also have their shortcomings. For example, interactive human feedback can be tedious.
Regarding the common pitfalls, although reward shaping (i.e. augment the natural reward function with more rewards) is often suggested as a way to improve the convergence of RL algorithms, [4] states that reward shaping (and progress estimators) should be used cautiously. If you want to perform reward shaping, you should probably be using potential-based reward shaping (which is guaranteed not to change the optimal policy).
The MathWorks' article Define Reward Signals discusses continuous and discrete reward functions (this is also discussed in [4]), and addresses some of their advantages and disadvantages.
Last but not least, the 2nd edition of the RL bible contains a section (17.4 Designing Reward Signals) completely dedicated to this topic.
Another similar question was also asked here.
If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with this reward function is that it's too sparse, meaning the agent has little guidance on how to modify their behavior to become better at attaining said goal, especially if the goal is hard to attain through a random policy in the first place (which is probably roughly what the agent starts with).
The practice of modifying the reward function to guide the learning agent is called reward shaping.
A good start is Policy invariance under reward transformations: Theory and application to reward shaping by Ng et al. The idea is to create a reward potential (see Theorem 1) on top of the existing reward. This reward potential should be an approximation of the true value of a given state. For instance, if you have a gridworld scenario where the goal is for the agent to reach some target square, you could create a reward potential based on the Manhattan distance to this target (without accounting for obstacles), which is an approximation to the true value of a given position.
Intuitively, creating a reward potential that is close to the true values makes the job easier for the learning agent because it reduces the disadvantage of being myopic, and the agent more quickly gets closer to a "somewhat good" policy from which it is easier to crawl toward the optimal policy.
Moreover, reward potentials have the property that they are consistent with the optimal policy. That is, the optimal policy to the true problem will not become suboptimal under the new, modified problem (with the new reward function).
