Return to Answer

Environment

This is the manifestation of the problem being solved. It might be a real physical situation (a road network and cars), or virtual on a computer (a board game on a computer). It includes all the machinery necessary to resolve what happens. E.g. in the real world the objects involved, how the agent exerts its control when taking actions, and the applicable real-world laws of physics. Or, in a simulated world, things like the rules of a board game, implemented in code.

State

This is the representation of a "position" at a certain time step within the environment. It may be something the agent can observe through sensors, or be provided directly by the computer system running a simulation.

For RL theory to hold, it is important that the state representation has the Markov Property, which is that the state accurately foretells the probabilities of rewards and following state for each action that could be taken. You do not need to know those probabilities in order to run RL algorithms (in fact that is a common case). However, it is important that the dependency between the state+action and what happens next holds reliably.

The state is commonly represented by a vector of values. These describe positions of pieces in a game, or positions and velocities of objects that have been sensed. A state may be built from observations, but does not have to match 1-to-1 with a single observation. Care must be taken to have enough information to have the Markov Property. So, for instance, a single image from a camera does not capture velocity - if velocity is important to your problem, you may need multiple consecutive images to build a useful state.

Model

In reinforcement learning, the term "model" specifically means a predictive model of the environment that resolves probabilities of next reward and next state following an action from a state. The model might be provided by the code for the environment, or it can be learned (separately to learning to behave in that environment).

Some RL algorithms can make use of a model to help with learning. Planning algorithms require one. So called "model free" algorithms do not because they do not make use of an explicit model, they work purely from experience.

There are broadly two types of model:

• A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

• A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

In more general stats/ML, the term "model" is more inclusive, and can mean any predictive system that you might build, not just predictions of next reward and state. However, the literature for RL typically avoids calling those "model", and uses terms like "function approximator" to avoid overloading the meaning of "model".

Environment

This is the manifestation of the problem being solved. It might be a real physical situation (a road network and cars), or virtual on a computer (a board game on a computer). It includes all the machinery necessary to resolve what happens. E.g. in the real world the objects involved, how the agent exerts its control when taking actions, and the applicable real-world laws of physics. Or, in a simulated world, things like the rules of a board game, implemented in code.

State

This is the representation of a "position" at a certain time step within the environment. It may be something the agent can observe through sensors, or be provided directly by the computer system running a simulation.

For RL theory to hold, it is important that the state representation has the Markov Property, which is that the state accurately foretells the probabilities of rewards and following state for each action that could be taken. You do not need to know those probabilities in order to run RL algorithms (in fact it is a common case that you don't know). However, it is important that the dependency between the state+action and what happens next holds reliably.

The state is commonly represented by a vector of values. These describe positions of pieces in a game, or positions and velocities of objects that have been sensed. A state may be built from observations, but does not have to match 1-to-1 with a single observation. Care must be taken to have enough information to have the Markov Property. So, for instance, a single image from a camera does not capture velocity - if velocity is important to your problem, you may need multiple consecutive images to build a useful state.

Model

In reinforcement learning, the term "model" specifically means a predictive model of the environment that resolves probabilities of next reward and next state following an action from a state. The model might be provided by the code for the environment, or it can be learned (separately to learning to behave in that environment).

Some RL algorithms can make use of a model to help with learning. Planning algorithms require one. So called "model free" algorithms do not because they do not make use of an explicit model, they work purely from experience.

There are broadly two types of model:

• A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

• A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

In more general stats/ML, the term "model" is more inclusive, and can mean any predictive system that you might build, not just predictions of next reward and state. However, the literature for RL typically avoids calling those "model", and uses terms like "function approximator" to avoid overloading the meaning of "model".

Environment

This is the manifestation of the problem being solved. It might be a real physical situation (a road network and cars), or virtual on a computer (a board game on a computer). It includes all the machinery necessary to resolve what happens. E.g. in the real world the objects involved, how the agent is controlled and the laws of physics. Or, in a simulated world, things like the rules of a board game.

State

This is the representation of a "position" at a certain time step within the environment. It may be something the agent can observe through sensors, or be provided directly by the computer system providing a simulation.

For RL theory to hold, it is important that the state representation has the Markov Property, which is that the state accurately foretells the probabilities of rewards and following state for each action that could be taken. You do not need to know those probabilities in order to run RL algorithms (in fact that is a common case). However, it is important the dependency between the state+action and what happens next holds reliably.

The state is commonly represented by a vector of values. These describe positions of pieces in a game, or positions and velocities of objects that have been sensed. A state may be build from observations, but does not have to match 1-to-1 with a single observation. Care must be taken to have enough information to have the Markov Property. So, for instance, a single image from a camera does not capture velocity - if velocity is important to your problem, you may need multiple consecutive images to build a useful state.

Model

In reinforcement learning, the term "model" specifically means a predictive model of the environment that can calculate probabilities next reward and next state following an action from a state. This might be provided by the code for the environment, or it can be learned (separately to learning to behave in that environment).

Some RL algorithms can make use of a model to help with learning. Planning algorithms require one. Others are often called "model free" as they do not make use of an explicit model, they work purely from experience.

There are broadly two types of model:

• A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

• A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

In more general stats/ML, the term "model" is more inclusive, and can mean any predictive system that you might build, not just predictions of next reward and state. However, the literature for RL typically avoids calling those "model", and uses terms like "function approximator" to avoid overloading the meaning of "model".

Environment

This is the manifestation of the problem being solved. It might be a real physical situation (a road network and cars), or virtual on a computer (a board game on a computer). It includes all the machinery necessary to resolve what happens. E.g. in the real world the objects involved, how the agent exerts its control when taking actions, and the applicable real-world laws of physics. Or, in a simulated world, things like the rules of a board game, implemented in code.

State

This is the representation of a "position" at a certain time step within the environment. It may be something the agent can observe through sensors, or be provided directly by the computer system running a simulation.

For RL theory to hold, it is important that the state representation has the Markov Property, which is that the state accurately foretells the probabilities of rewards and following state for each action that could be taken. You do not need to know those probabilities in order to run RL algorithms (in fact that is a common case). However, it is important that the dependency between the state+action and what happens next holds reliably.

The state is commonly represented by a vector of values. These describe positions of pieces in a game, or positions and velocities of objects that have been sensed. A state may be built from observations, but does not have to match 1-to-1 with a single observation. Care must be taken to have enough information to have the Markov Property. So, for instance, a single image from a camera does not capture velocity - if velocity is important to your problem, you may need multiple consecutive images to build a useful state.

Model

In reinforcement learning, the term "model" specifically means a predictive model of the environment that resolves probabilities of next reward and next state following an action from a state. The model might be provided by the code for the environment, or it can be learned (separately to learning to behave in that environment).

Some RL algorithms can make use of a model to help with learning. Planning algorithms require one. So called "model free" algorithms do not because they do not make use of an explicit model, they work purely from experience.

There are broadly two types of model:

• A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

• A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

In more general stats/ML, the term "model" is more inclusive, and can mean any predictive system that you might build, not just predictions of next reward and state. However, the literature for RL typically avoids calling those "model", and uses terms like "function approximator" to avoid overloading the meaning of "model".

Environment

This is the manifestation of the problem being solved. It might be a real physical situation (a road network and cars), or virtual on a computer (a board game on a computer). It includes all the machinery necessary to resolve what happens. E.g. in the real world the objects involved, how the agent is controlled and the laws of physics. Or, in a simulated world, things like the rules of a board game.

State

This is the representation of a "position" at a certain time step within the environment. It may be something the agent can observe through sensors, or be provided directly by the computer system providing a simulation.

For RL theory to hold, it is important that the state representation has the Markov Property, which is that the state accurately foretells the probabilities of rewards and following state for each action that could be taken. You do not need to know those probabilities in order to run RL algorithms (in fact that is a common case).

The state is commonly represented by a vector of values. These describe positions of pieces in a game, or positions and velocities of objects that have been sensed. A state may be build from observations, but does not have to match 1-to-1 with a single observation. Care must be taken to have enough information to have the Markov Property. So, for instance, a single image from a camera does not capture velocity - if velocity is important to your problem, you may need multiple consecutive images to build a useful state.

Model

In reinforcement learning, the term "model" specifically means a predictive model of the environment that can calculate probabilities next reward and next state following an action from a state. This might be provided by the code for the environment, or it can be learned (separately to learning to behave in that environment).

Some RL algorithms can make use of a model to help with learning. Planning algorithms require one. Others are often called "model free" as they do not make use of an explicit model, they work purely from experience.

In more general stats/ML, the term "model" is more inclusive, and can mean any predictive system that you might build, not just predictions of next reward and state. However, the literature for RL typically avoids calling those "model", and uses terms like "function approximator" to avoid overloading the meaning of "model".

Environment

This is the manifestation of the problem being solved. It might be a real physical situation (a road network and cars), or virtual on a computer (a board game on a computer). It includes all the machinery necessary to resolve what happens. E.g. in the real world the objects involved, how the agent is controlled and the laws of physics. Or, in a simulated world, things like the rules of a board game.

State

This is the representation of a "position" at a certain time step within the environment. It may be something the agent can observe through sensors, or be provided directly by the computer system providing a simulation.

For RL theory to hold, it is important that the state representation has the Markov Property, which is that the state accurately foretells the probabilities of rewards and following state for each action that could be taken. You do not need to know those probabilities in order to run RL algorithms (in fact that is a common case). However, it is important the dependency between the state+action and what happens next holds reliably.

The state is commonly represented by a vector of values. These describe positions of pieces in a game, or positions and velocities of objects that have been sensed. A state may be build from observations, but does not have to match 1-to-1 with a single observation. Care must be taken to have enough information to have the Markov Property. So, for instance, a single image from a camera does not capture velocity - if velocity is important to your problem, you may need multiple consecutive images to build a useful state.

Model

In reinforcement learning, the term "model" specifically means a predictive model of the environment that can calculate probabilities next reward and next state following an action from a state. This might be provided by the code for the environment, or it can be learned (separately to learning to behave in that environment).

Some RL algorithms can make use of a model to help with learning. Planning algorithms require one. Others are often called "model free" as they do not make use of an explicit model, they work purely from experience.

There are broadly two types of model:

• A distribution model which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.

• A sampling model which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.

In more general stats/ML, the term "model" is more inclusive, and can mean any predictive system that you might build, not just predictions of next reward and state. However, the literature for RL typically avoids calling those "model", and uses terms like "function approximator" to avoid overloading the meaning of "model".