# How to define a reward function in POMDPs?

How do I define a reward function for my POMDP model?

In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have to do define a reward for every possible action-state combination. I think that the examples in the literature might not be practical in reality, but only for the purpose of explanation.

There is no major difference here between a POMDP and MDP. When setting reward values, you are generally trying to give the minimal information to the agent that when the sum of rewards is maximised, it solves the problem that you are posing.

In literature it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have do define a reward for every possible Action-State combination.

Some defined value of reward has to be returned after all state, action pairs taken in the environment. The value could be $$0$$ of course.

Reward can depend on, or be a function of, current state, action, next state and a random factor. In a POMDP it may also be from any unobserved factor in the environment (you might know this in a simulation because you have created the environment and are choosing not to share the data with the agent).

In practice, the reward often does not have to depend on all the possible factors. In addition, the relationship between the factors and the possible reward can be very simple or sparse.

Classic examples you may find in the literature include:

• Reward in a game can be simply $$+1$$ for winning, or $$-1$$ for losing, granted at the end. All other rewards are $$0$$

• If your goal is to reach a certain state, such as exit from a maze, in minimum time, then a fixed reward of $$-1$$ per time step is enough to express the need to minimize the total number of steps.

• For a goal of maintaining stability and avoiding a failure in e.g. CartPole, then a reward of +1 per time step without failure suffices.

All of these examples express the reward function as a simple condition plus one or two numbers. The key thing is that they allow you to calculate a suitable reward on each and every time step. They are all strictly reward functions - the general case can cover very complex functions if you wish, that will depend entirely on the goals you want to set the agent to learn/solve.

In a POMDP, you minimize the following cost function

\begin{align} J_{\pi_{0:N-1}} (\cdot) &= \mathbb{E} \left( g_N (x_N) + \sum_{k=0}^{N-1} g_k (x_k, \pi_k(\cdot)) \right), \end{align} where $$g_N$$ is the terminal cost and $$g_k$$ is the step cost.

Note that this is only the formal problem definition. If you talk about an actual system, you might have a hard time recognizing that this is connected to your reward. What you observe is formally a value of the function J and practically a simple float

More details in German are on my blog: https://martin-thoma.com/probabilistische-planung/#mdp-vs-pomdp-vs-rl_1