7
$\begingroup$

I'm studying reinforcement learning. It seems that "state" and "observation" mean exactly the same thing. They both capture the current state of the game.

Is there a difference between the two terms? Is the observation maybe the state after the action has been taken?

$\endgroup$
8
$\begingroup$

Sometimes observation and state overlap completely, which is convenient. However, there is no reason to expect it in all cases, and that's where interesting problems occur.

Reinforcement learning theory is based on Markov Decision Processes. This leads to a formal definition of state. Most importantly, the state must have the Markov property. Which means that for RL to work according to theory, that knowing the state means that you know everything knowable that could determine the response of the environment to a specific action. Everything that remains must be purely stochastic and unknowable in principle until after the action is resolved.

Systems like deterministic or probability-driven games, and computer-controlled simulations can be designed to have easily observable states that have this property. Games with this trait are often called "games of perfect information", although you may have unknown information, provided it is revealed in a purely stochastic manner.

In practice, real world interactions contain far too much detail for any observation to be a true state with the Markov property. For instance, consider the inverted pendulum environment, a classic RL toy problem. A real inverted pendulum would behave differently depending on its temperature, which could vary along its length. The joint and actuators might be sticky. Rotations and movement will alter temperature and friction, etc. However, a RL agent will typically only consider current motion and position of the trolley and pendulum. In this case, the observation of 4 traits is usually good enough, and a state based on this almost has the Markov property.

There are also problems where observations are not enough to make usable state data for a RL system. The Deep Mind Atari DQN paper had examples of a couple of these. The first example is that a single frame lost data about motion. This could be addressed by taking four consecutive frames and combining them to make a single state. It could be argued that each frame is an observation, and that four observations had to be combined in order to construct a more useful state (although this could be put aside as just semantics).

The second example in Atari DQN is that the pixel observations did not include data that the game was tracking but that was not visible on screen. Games with large scrolling maps are a weakness of the Atari-playing DQN, because its state has no memory of screens other than the four used for movement. An example of such a game, where Deep Mind's player did much worse than a human player is Montezuma's Revenge, where to progress it is necessary to remember some off-screen locations.

There are ways to address knowledge that there is unobserved but relevant state in a problem. The general framework for describing the problem is Partially Observable Markov Decision Processes (POMDPs). Workable solutions include adding explicit memory or "belief state" to the state representation, or using a system such as RNN in order to internalise the learning of a state representation driven by a sequence of observations.

$\endgroup$
  • $\begingroup$ Thanks for the response! So an observation is a subset from the state? $\endgroup$ – echo Apr 9 '18 at 21:58
  • 1
    $\begingroup$ @echo: Not necessarily. You could also observe things that are irrelevant to the state. So it is more accurate to say that state and observation may overlap. Generally, when they overlap a lot, RL is easier to apply, so you try and make this happen - most toy problems in RL are designed so that observation and state overlap perfectly, for convenience. But sometimes interesting problems require you to deal with missing/unobserved state data. $\endgroup$ – Neil Slater Apr 9 '18 at 22:03
  • $\begingroup$ In a few words, an "observation" is a term used to indicate what the agent observes (or not) from the environment (at every time step). An observation can e.g. be a frame of a game. We are particularly interested in observations when states are not completely or easily specified or do not have the Markov property. In that case, an observation may be useful to build a state. POMDPs are a formal way of modelling problems where states do not e.g. have the Markov property. So, in POMDPs, there is the concept of observation in order to cope with the non-Markovian or not completely specified states. $\endgroup$ – nbro Feb 10 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.