Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I'm studying reinforcement learning. It seems that "state" and "observation" mean exactly the same thing. They both capture the current state of the game.
Is there a difference between the two terms? Is the observation maybe the state after the action has been taken?
Sometimes observation and state overlap completely, which is convenient. However, there is no reason to expect it in all cases, and that's where interesting problems occur.
Reinforcement learning theory is based on Markov Decision Processes. This leads to a formal definition of state. Most importantly, the state must have the Markov property. Which means that for RL to work according to theory, that knowing the state means that you know everything knowable that could determine the response of the environment to a specific action. Everything that remains must be purely stochastic and unknowable in principle until after the action is resolved.
Systems like deterministic or probability-driven games, and computer-controlled simulations can be designed to have easily observable states that have this property. Games with this trait are often called "games of perfect information", although you may have unknown information, provided it is revealed in a purely stochastic manner.
In practice, real world interactions contain far too much detail for any observation to be a true state with the Markov property. For instance, consider the inverted pendulum environment, a classic RL toy problem. A real inverted pendulum would behave differently depending on its temperature, which could vary along its length. The joint and actuators might be sticky. Rotations and movement will alter temperature and friction, etc. However, a RL agent will typically only consider current motion and position of the trolley and pendulum. In this case, the observation of 4 traits is usually good enough, and a state based on this almost has the Markov property.
There are also problems where observations are not enough to make usable state data for a RL system. The Deep Mind Atari DQN paper had examples of a couple of these. The first example is that a single frame lost data about motion. This could be addressed by taking four consecutive frames and combining them to make a single state. It could be argued that each frame is an observation, and that four observations had to be combined in order to construct a more useful state (although this could be put aside as just semantics).
The second example in Atari DQN is that the pixel observations did not include data that the game was tracking but that was not visible on screen. Games with large scrolling maps are a weakness of the Atari-playing DQN, because its state has no memory of screens other than the four used for movement. An example of such a game, where Deep Mind's player did much worse than a human player is Montezuma's Revenge, where to progress it is necessary to remember some off-screen locations.
There are ways to address knowledge that there is unobserved but relevant state in a problem. The general framework for describing the problem is Partially Observable Markov Decision Processes (POMDPs). Workable solutions include adding explicit memory or "belief state" to the state representation, or using a system such as RNN in order to internalise the learning of a state representation driven by a sequence of observations.
