2
$\begingroup$

I was reading in the article A tutorial on partially observable Markov decision processes (p. 120), by Michael L. Littman, that $\sum_{z \in Z}O(a, s',z) =1$, where $a$ is the action, $s'$ the next possible state and $z$ a certain/specific observation.

How come that the observation function $O(a, s', z)$ adds up to $1$ in POMDP?

$\endgroup$
2
$\begingroup$

$O(a, s', z) = \mathbb{P}(z \mid a, s')$ is a conditional probability distribution, so it always needs to sum up to $1$. You should interpret $O(a, s', z)$ as the probability of observation $z$, given that the agent took action $a$ and landed in state $s'$.

$O(a, s', z)$ is thus not a joint distribution, even though the notation $O(a, s', z)$ might suggest it. In this case, $O(a, s', z)$ simply means that $O$ is a function of $a$, $z$ and $s'$.

If you want to see a proof that conditional probability distributions sum up to 1, have a look at this post.

$\endgroup$
  • $\begingroup$ But does it still mean, that I have to observe a state somehow, even the information might not be accurate? $\endgroup$ – Bryan McGill Mar 28 at 17:22
  • 1
    $\begingroup$ @BryanMcGill You don't observe any state (in POMDPs). You get "observations", which you use to model your belief in which state you might be in. $\endgroup$ – nbro Mar 28 at 17:25
  • $\begingroup$ That clarifies everythinga. I came along some illustrations where "observations" were connected to states and I got confused. Maybe you could edit your answer regarding that one sentence in the beginning. $\endgroup$ – Bryan McGill Mar 28 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.