3
$\begingroup$

I am new to RL and I am following Sutton & Barto's book.

My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. However, I think that the policy should be defined in terms of observations and not states because I think it is not always possible for an agent to fully capture the state due to various reasons, maybe lack of sensors or lack of memory.

So, why are policies defined as functions of states and not observations?

Improve this question
$\endgroup$
5
  • $\begingroup$ From what I remember, RL can handle POMDP. $\endgroup$
    – FourierFlux
    Sep 19, 2021 at 4:47
  • $\begingroup$ oh, I'll read about it, because I think one of the reason they defined it that way can be that, they mentioned the environment to be fully observable. But then how my agent is going to make actions? if i dont know what state I am in. It is all very confusing in my head @FourierFlux $\endgroup$
    – hakiki_makato
    Sep 19, 2021 at 5:10
  • 1
    $\begingroup$ From what I remember something in Sutton's book says function based approximations can work on POMDP, but discrete state cannot. I would need to look it up though to be sure. $\endgroup$
    – FourierFlux
    Sep 19, 2021 at 5:19
  • $\begingroup$ To be honest though it shouldn't work intuitively speaking, since an RL problem should ultimately be a MDP and there cannot be hidden states which have memory so I do not know. $\endgroup$
    – FourierFlux
    Sep 19, 2021 at 5:22
  • $\begingroup$ @FourierFlux: I think you have remembered correctly. The key is that there are different degrees and types of missing knowledge that can impact an agent learning to act in an MDP. I think a good answer will try to cover that, as well as how observatons and state can interact. $\endgroup$
    – Neil Slater
    Sep 19, 2021 at 8:58

1 Answer 1

Reset to default
4
$\begingroup$

Ultimately, a policy must be such that is is possible for an agent to execute it.

If the policy depends on the state, the implicit assumption is that the agent has knowledge of the state and can therefore choose its actions accordingly. This is the common case of a MDP as an underlying framework for RL.

If the state is not known to the agent, it may instead perceive observations that have some relation to the state (although they might not fully reveal the state). Then, one can condition policies on the last received observation.

However, it is useful to note that conditioning on the last observation in a partially observable setting is in general not sufficient for acting optimally. There are cases where one may need to remember a longer history of observations to decide which action is best. In general, acting optimally in such a partially observable setting requires that the policy is a function of the complete history of past actions taken and observations perceived. The underlying framework is then the partially observable MDP, or POMDP.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is a good answer, but one thing is missing. For example, in DQN applied to Atari games, are you going to learn a policy from states to actions or from observations to actions? In DQN, you're basically applying Q-learning with function approximation to POMDPs, which you assume to be MDPs by approximating states as a sequence of, say, 4 observations. So, in this case, will you use this stack of observations as an approximation to the current state even after having stopped learning, so will the policy, effectively, be a mapping from a sequence of successive observations to actions? $\endgroup$
    – nbro
    Sep 22, 2021 at 12:50
  • $\begingroup$ @nbro In an MDP, the current state and action are sufficient to predict the next state, and knowing any older past states will not help to predict better. The "trick" in DQN uses a stack of frames and calls it "the state" and then the problem is solved like it was an MDP. But the stack of frames is not a state in the MDP sense above: it's not the only thing besides the action that matters for the next observation stack. In an Atari game, an MDP state could be the state of the internal game memory. That and the current input determine the next memory state. Trick in DQN is an approximation. $\endgroup$
    – mikkola
    Sep 22, 2021 at 14:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .