2
$\begingroup$

How do I define a reward function for my POMDP model?

In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have to do define a reward for every possible action-state combination. I think that the examples in the literature might not be practical in reality, but only for the purpose of explanation.

$\endgroup$
1
$\begingroup$

There is no major difference here between a POMDP and MDP. When setting reward values, you are generally trying to give the minimal information to the agent that when the sum of rewards is maximised, it solves the problem that you are posing.

In literature it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have do define a reward for every possible Action-State combination.

Some defined value of reward has to be returned after all state, action pairs taken in the environment. The value could be $0$ of course.

Reward can depend on, or be a function of, current state, action, next state and a random factor. In a POMDP it may also be from any unobserved factor in the environment (you might know this in a simulation because you have created the environment and are choosing not to share the data with the agent).

In practice, the reward often does not have to depend on all the possible factors. In addition, the relationship between the factors and the possible reward can be very simple or sparse.

Classic examples you may find in the literature include:

  • Reward in a game can be simply $+1$ for winning, or $-1$ for losing, granted at the end. All other rewards are $0$

  • If your goal is to reach a certain state, such as exit from a maze, in minimum time, then a fixed reward of $-1$ per time step is enough to express the need to minimize the total number of steps.

  • For a goal of maintaining stability and avoiding a failure in e.g. CartPole, then a reward of +1 per time step without failure suffices.

All of these examples express the reward function as a simple condition plus one or two numbers. The key thing is that they allow you to calculate a suitable reward on each and every time step. They are all strictly reward functions - the general case can cover very complex functions if you wish, that will depend entirely on the goals you want to set the agent to learn/solve.

$\endgroup$
  • $\begingroup$ There is a big difference between not defining the reward function for some points and defining it as 0. Did you really want to write that it has not to be defined? $\endgroup$ – Martin Thoma Oct 30 '18 at 15:15
  • $\begingroup$ @MartinThoma: I wrote "Reward does need to be defined for all actions and states" I don't see anywhere I said that it does not need to be defined (if there is somewhere I have stated that, then I cannot see where and would appreciate it pointed out). A value of $0$ is of course defined. $\endgroup$ – Neil Slater Oct 30 '18 at 15:25
  • $\begingroup$ I think the reward needs to be defined in all possible state/action pairs. It is, however, fine if it is 0. But not defined means your complete problem statement is problematic. Or could you give a concrete example where it would be acceptable? $\endgroup$ – Martin Thoma Oct 30 '18 at 16:10
  • $\begingroup$ @MartinThoma: I don't say that "not defined is acceptable" in this answer. I say the opposite - that reward must take a defined value for all time steps. Are you seeing an extra "not" in any of my sentences? $\endgroup$ – Neil Slater Oct 30 '18 at 16:21
  • $\begingroup$ Ooops, sorry. Yes, seems like I hallucinated an extra "not" 🙈. Sorry for the confusion. $\endgroup$ – Martin Thoma Oct 30 '18 at 16:28
0
$\begingroup$

In a pomdp, you minimize

$\begin{align} J_{\pi_{0:N-1}} (\cdot) := & \mathbb{E} (g_N (x_N) \\ + & \sum_{k=0}^{N-1} g_k (x_k, \pi_k(\cdot))) \end{align}$

Where g_N is the terminal cost and g_k is the step cost.

Note that this is only the formal problem definition. If you talk about an actual system, you might have a hard time to recognizes that this is connected to your reward. What you observe is formally a value of the function J and practically a simple float

More details in German are on my blog: https://martin-thoma.com/probabilistische-planung/#mdp-vs-pomdp-vs-rl_1

$\endgroup$
  • $\begingroup$ What does J refer to? Belief state? $\endgroup$ – Bryan McGill Oct 30 '18 at 15:02
  • $\begingroup$ J is just a name. It is the cost function. You can also call it $f$ $\endgroup$ – Martin Thoma Oct 30 '18 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.