# How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the game is to reach the finishing line before the opponent.

While working on this problem, I've realized that we can almost forget about the presence of our opponent and just focus on getting the agent to the finish line as quickly as possible.

I started with a simple

• $$-1$$ reward for every timestep
• $$+100$$ reward for winning, and
• $$-100$$ for losing.

When I was experimenting with this, I felt like the rewards may be too sparse, as my agent was converging to pretty poor average returns. I iterated to a function of speed and distance travelled (along with the $$+100$$ reward), but, after some experimentation, I started feeling like the agent might be able to achieve high returns without necessarily being the fastest to the finish line.

I'm thinking that I return to the first approach and possibly add in some reward for being in the first place (as a function of the opponent's distance behind the agent).

What else could I try? Should I try and spread the positive rewards out more for good behavior? should I create additional rewards/penalties for perhaps hitting obstacles and using boosts or can I expect the agent to learn the correlation?

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of $$1$$ for winning, $$0$$ for losing, and $$0$$ at all other time steps seems to fit that narrative. If a draw is identical to a loss, then it should provide reward $$0$$; otherwise, a reward of $$0.5$$ seems reasonable. These rewards provide model interpretability: an expected return of $$p$$ (estimated with a state-value or action-value) at a certain state under the current policy would signify a $$p$$ chance of winning. Also, keeping the rewards in absolute value at most 1 can aid in training speed and prevent divergence, but it often isn't necessary for deep reinforcement learning problems. You most certainly can add other rewards based on partial progress towards the goal, but as it seems you found out, they may lead to incorrect results.
Depending on your access to the environment internals, you may need other analogous approaches. For example, you could initialize the agents at the original starting line (i.e. not partway down the map) and then see which agent makes it $$n$$ units down the map first to determine the winner. Once a winning policy is learned by the RL agent, then gradually increase $$n$$ until $$n$$ matches the distance from the starting line to the finish line. Since it seems like you have distance traveled and distance to the opponent as features, you may instead try this method if you are unable to initialize the agents wherever you want on the map and instead can only initialize them on the starting line.