Designing reward functions
Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any other situation).
However, in certain cases, the specification of the reward function can be a difficult task [1, 2, 3] because there are many (often unknown) factors that could affect the performance of the RL agent. For example, consider the driving task, i.e. you want to teach an agent to drive e.g. a car. In this scenario, there are so many factors that affect the behavior of a driver. How can we incorporate and combine these factors in a reward function? How do we deal with unknown factors?
So, often, designing a reward function is a trial-and-error and engineering process (so there is no magic formula that tells you how to design a reward function in all cases). More precisely, you define an initial reward function based on your knowledge of the problem, you observe how the agent performs, then tweak the reward function to achieve greater performance. For example, if you have trained an RL agent to play chess, maybe you observed that the agent took a lot of time to converge (i.e. find the best policy to play the game), so you could design a new reward function that penalizes the agent for every non-win move (maybe it will hurry up!)
Of course, this trial-and-error approach is not ideal, and it can sometimes be impractical (because maybe it takes a lot of time to train the agent) and lead to misspecified reward signals.
Misspecification of rewards
It is well known that the misspecification of the reward function can have unintended and even dangerous consequences . To overcome the misspecification of rewards or improve the reward functions, you have some options, such as
Learning from demonstrations (aka apprenticeship learning), i.e. do not specify the reward function directly, but let the RL agent imitate another agent's behavior, either to
- learn the policy directly (known as imitation learning ), or
- learn a reward function first to later learn the policy (known as inverse reinforcement learning  or sometimes known as reward learning)
Incorporate human feedback  in the RL algorithms (in an interactive manner)
Transfer the information in the policy learned in another but similar environment to your environment (i.e. use some kind of transfer learning for RL )
Of course, these solutions or approaches can also have their shortcomings. For example, interactive human feedback can be tedious.
Regarding the common pitfalls, although reward shaping (i.e. augment the natural reward function with more rewards) is often suggested as a way to improve the convergence of RL algorithms,  states that reward shaping (and progress estimators) should be used cautiously. If you want to perform reward shaping, you should probably be using potential-based reward shaping (which is guaranteed not to change the optimal policy).
The MathWorks' article Define Reward Signals discusses continuous and discrete reward functions (this is also discussed in ), and addresses some of their advantages and disadvantages.
Last but not least, the 2nd edition of the RL bible contains a section (17.4 Designing Reward Signals) completely dedicated to this topic.