I stumbled across observations which is what the RL sees which is also inputs. At first, I just threw it off saying they were the same thing

For most toy environments used to teach yourself RL, this is true. It's because the designers of those toy environments made sure it was the case. The relationship between observation and state adds a level of complexity you don't need when starting out.

In addition, for many classic environments, such as skill-based board games, it is also true that whatever the agent can observe at the current moment is enough to reliably predict the probabilities - and this learn values - of all immediate outcomes that depend on action.

If you are designing a real-world system, and can collect data at a single point in time that makes a reliable state, then this is a good idea, and you should do that. Again in that case, state and observation would effectively be the same thing.

However, the state is what the agent believes is a good representation of what it sees

The state is ideally a summary of history that can reliably be used to predict the immediate future - or at least its distribution. This is the Markov property, and is assumed in all the maths behind the MDP model of basic reinforcement learning.

Whn you get into the agent's beliefs, you are typically considering a partially observable state (for a POMDP), and this is one way that you can try to bridge the gap between what the agent can actually gather data about, and its need to predict the future behaviour.

How does the RL decide this though? How does it convert the observations into a state?

There is no single approach. This is an analysis and engineering issue that depends on the problem at hand. You have to start with:

• What can the agent observe (a physical or logical constraint that depends on what the agent is and what the environment is)?
• Is there an obvious set of features that the agent does not observe directly, but needs to know in order to have a state that at least approximately has the Markov property?
• Is there a way to logically construct any additional features from the observation or observation history?
  • If this can be done, then usually it is hand-coded by the developer, converting observation history to state data. So the agent doesn't "know" so much as the problem is solved for it beforehand.
  • An example: For Atari DQN, an observation is a screengrab of current pixels. Some of these pixels represent game objects that are moving, and to predict what happens next it is important to know direction and speed of movement. However, the single screengrab does not show the movement. By stacking multiple recent frames (four for the original DQN project) together it is possible to represent movement in the state.
• Is there a known underlying model of the environment, where the agent can start with a guess at the full state, and refine it?
  • If yes, this can be modelled as a POMDP. This is more flexible than the hard-coded fixup, the agent really is learning to update its beliefs, but there has to be a well-understood belief system for the hidden state which the agent is told how to manipulate in advance.
  • An example: In a game of whist where the agent cannot see opponent's hands, it can guess distributions of cards, and modify its guesses during play. For instance if one player cannot follow suit, then it can immediately adjust its knowledge of probability distribution of remaining cards in that suit amongst the other players.
• Are there still some unknown internal states, but a summary of history so far might give some clues?
  • If yes, you can hope to convert the observation history to a useful summary automatically by giving the agent access to an arbitrary set of features and a model that can learn them along with the value predictions. This is typically some form of recurrent neural network (RNN) like LSTM model.
  • An example: This is an alternative way for the agent to learn about rules for moving objects. So in the Atari DQN, instead of giving it stacks of screengrabs, you can change its model to LSTM and only feed it the raw observations of individual screengrabs. This is a harder learning task for the agent, but in some ways more satisfying as once you figure out the right archtecture to do this you have an agent that learns to use image history by itself to predict values. It may even learn object behaviour that has historic "tells" longer than the stacked version and be able to take advantage of that knowledge - e.g. an enemy piece that can go invisible for a second and reappear in the same place.

As you can see, this can get quite complex. Which is why the difference between observation and state is often put to one side for learning materials about basic RL.