How does the agent derive the state from the observations in Reinforcement Learning?

Question

I am not at all experienced in RL programs. I have been reading up about them recently, and I learned about states. I thought of it as all the inputs to the RL. Then I stumbled across observations which is what the RL sees which is also inputs. At first, I just threw it off saying they were the same thing, but then I searched it up and apparently it is not. Apparently, observations is what the agent can physically see. However, the state is what the agent believes is a good representation of what it sees. How does the RL decide this though? How does it convert the observations into a state?

If you respond to this question, please include a simple example. It makes it MUCH easier to understand.

Answer 1

I stumbled across observations which is what the RL sees which is also inputs. At first, I just threw it off saying they were the same thing

For most toy environments used to teach yourself RL, this is true. It's because the designers of those toy environments made sure it was the case. The relationship between observation and state adds a level of complexity you don't need when starting out.

In addition, for many classic environments, such as skill-based board games, it is also true that whatever the agent can observe at the current moment is enough to reliably predict the probabilities - and this learn values - of all immediate outcomes that depend on action.

If you are designing a real-world system, and can collect data at a single point in time that makes a reliable state (as in you can predict the distribution of immediate outcomes accurately from it), then this is a good idea, and you should do that. Again in that case, state and observation would effectively be the same thing.

However, the state is what the agent believes is a good representation of what it sees

The state is ideally a summary of history that can reliably be used to predict the immediate future - or at least its distribution. This is the Markov property, and is assumed in all the maths behind the MDP model of basic reinforcement learning.

Whn you get into the agent's beliefs, you are typically considering a partially observable state (for a POMDP), and this is one way that you can try to bridge the gap between what the agent can actually gather data about, and its need to predict the future behaviour.

How does the RL decide this though? How does it convert the observations into a state?

There is no single approach. This is an analysis and engineering issue that depends on the problem at hand. You have to start with:

• What can the agent observe (a physical or logical constraint that depends on what the agent is and what the environment is)?
• Is there an obvious set of features that the agent does not observe directly, but needs to know in order to have a state that at least approximately has the Markov property?
• Is there a way to logically construct any additional features from the observation or observation history?
  • If this can be done, then usually it is hand-coded by the developer, converting observation history to state data. So the agent doesn't "know" so much as the problem is solved for it beforehand.
  • An example: For Atari DQN, an observation is a screengrab of current pixels. Some of these pixels represent game objects that are moving, and to predict what happens next it is important to know direction and speed of movement. However, the single screengrab does not show the movement. By stacking multiple recent frames (four for the original DQN project) together it is possible to represent movement in the state.
• Is there a known underlying model of the environment, where the agent can start with a guess at the full state, and refine it?
  • If yes, this can be modelled as a POMDP. This is more flexible than the hard-coded fixup, the agent really is learning to update its beliefs, but there has to be a well-understood belief system for the hidden state which the agent is told how to manipulate in advance.
  • An example: In a game of whist where the agent cannot see opponent's hands, it can guess distributions of cards, and modify its guesses during play. For instance if one player cannot follow suit, then it can immediately adjust its knowledge of probability distribution of remaining cards in that suit amongst the other players.
• Are there still some unknown internal states, but a summary of history so far might give some clues?
  • If yes, you can hope to convert the observation history to a useful summary automatically by giving the agent access to an arbitrary set of features and a model that can learn them along with the value predictions. This is typically some form of recurrent neural network (RNN) like LSTM model.
  • An example: This is an alternative way for the agent to learn about rules for moving objects. So in the Atari DQN, instead of giving it stacks of screengrabs, you can change its model to LSTM and only feed it the raw observations of individual screengrabs. This is a harder learning task for the agent, but in some ways more satisfying as once you figure out the right archtecture to do this you have an agent that learns to use image history by itself to predict values. It may even learn object behaviour that has historic "tells" longer than the stacked version and be able to take advantage of that knowledge - e.g. an enemy piece that can go invisible for a second and reappear in the same place.

As you can see, this can get quite complex. Which is why the difference between observation and state is often put to one side for learning materials about basic RL.

Answer 2

State - the true state of the environment
Observation - what the agent can actually see (observe)

Take for example some Atari game, e.g. Pacman. At any given moment the true state would contain the coordinates of the positions of Pacman, the ghosts, the food pallets, the walls of the maze, and maybe other stuff. The obaervation however is just an image of pixels, nothing else. What is more important, you have no idea what the actions do, i.e. you cannot predict the next observation given the current observation and the selected action. The only thing you can do is run that action in the environment and see what happens.

The most famous RL algorithms like policy gradient and dqn are so-called model-free algorithms. Your agent can learn how to select actions only from observations and interaction. The agent does not need the true state of the environment or the mechanics of the world.

There are also environments where you actually observe the true state. For example in chess you can see the entire board, there is no hidden information. Most probably if you know the true state then you also know the mechanics, i.e. you can predict the next state. In this case you don't have to use model-free RL. Search algorithms like A* or MCTS would probably produce much better results.

Finally, you can try and learn the state of the environment and the mechanics of the world from observations and interactions. Once you have those you can apply search algorithms to learn how to act. Here things get quite complicated and I would refer you to this post:
https://bair.berkeley.edu/blog/2019/12/12/mbpo/

For a beginner I would suggest getting very comfortable with the first two approaches before moving on.