Skip to main content
24 votes
Accepted

How to define states in reinforcement learning?

The problem of state representation in Reinforcement Learning (RL) is similar to problems of feature representation, feature selection and feature engineering in supervised or unsupervised learning. ...
Neil Slater's user avatar
  • 32.1k
12 votes

How to define states in reinforcement learning?

A common early approach to modeling complex problems was discretization. At a basic level, this is splitting a complex and continuous space into a grid. Then you can use any of the classic RL ...
John Doucette's user avatar
7 votes
Accepted

What are the state space and the state transition function in AI?

Initial state How things are at first. In your particular example, it would be where your k knights are placed on the board initially. Your problem doesn't ...
Keno's user avatar
  • 575
6 votes
Accepted

What is the relation between the context in contextual bandits and the state in reinforcement learning?

The notion of a state in reinforcement learning is (more or less) the same as the notion of a context in contextual bandits. The main difference is that, in reinforcement learning, an action $a_t$ in ...
nbro's user avatar
  • 40.6k
6 votes
Accepted

What techniques are used to make MDP discrete state space manageable?

tl:dr Read chapter 9 of an Introduction of Reinforcement Learning There is definitely a problem (a curse if you will) when the dimensionality of a task (MDP) grows. For fun, lets extend your problem ...
Jaden Travnik's user avatar
5 votes

What constitutes a large space state (in Q-learning)?

I know this might be specific to different problems but does anyone know if there is any rule of thumb or references on what constitutes a large state space? Not really, it is all relative. There are ...
Neil Slater's user avatar
  • 32.1k
4 votes
Accepted

What is a high dimensional state in reinforcement learning?

Usually when people write about having a high-dimensional state space, they are referring to the state space actually used by the algorithm. Suppose my state is a high dimensional vector of $N$ ...
Dennis Soemers's user avatar
  • 10.3k
4 votes

What exactly are partially observable environments?

You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

What is the difference between terminal state, nonterminal states and normal states?

Terminal state is always the same in the sense that it represents the same thing, that the episode is over. They don’t need to be the exact same state; for instance you could have an $n$ by $n$ grid ...
David's user avatar
  • 4,910
3 votes

Is it appropriate to represent 'total failure' as an absorbing state?

In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

What exactly are partially observable environments?

First, note that the current state does not determine the next state. What determines the next state are the dynamics of the environment, which, in the context of reinforcement learning and, in ...
nbro's user avatar
  • 40.6k
3 votes

What is the relation between the context in contextual bandits and the state in reinforcement learning?

Conceptually, in general, how is the context being handled in CB, compared to states in RL? In terms of its place in the description of Contextual Bandits and Reinforcement Learning, context in CB is ...
Neil Slater's user avatar
  • 32.1k
3 votes

It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

A stateless RL problem can be reduced to a Multiarmed Bandit (MAB) problem. In such a scenario, taking an action will not change the state of the agent. So, this is the setting of a conventional MAB ...
Borhan Kazimipour's user avatar
3 votes
Accepted

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

How does replacing states with latent representations help RL agents?

In short, it is much easier for the agent to learn from a smaller dimensional state space. This is because the agent must also do representation learning; i.e. it must also infer what the state is ...
David's user avatar
  • 4,910
3 votes

Implementing an RL agent on a variable action space

I will try to answer your questions as best I can, potentially building on some of your ideas. Please note that I'm making certain assumptions about how to design the action space because I don't have ...
Cesar Ruiz's user avatar
3 votes
Accepted

What kind of observation state would you give for that environment?

Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem ...
Neil Slater's user avatar
  • 32.1k
2 votes

What is a high dimensional state in reinforcement learning?

Yes, it makes sense to use DQN in state space with small number of dimensions as well. It doesn't really matter how big your state dimension is, but if you have state with 2 dimensions for instance ...
Brale's user avatar
  • 2,386
2 votes
Accepted

What happens when the agent faces a state that never before encountered?

Having too many states to actually visit is a common problem in RL. This is exactly why we often use function approximation. If you replace your q table with a good function approximator such as a ...
chessprogrammer's user avatar
2 votes

How to approach a blackjack-like card game with the possibility of cards being counted?

How can I approach projects with a big state space without loosing a huge chunk of predictability (which I might fear with DQN, DDPQ or TD3)? You can impact this by choosing a combination of function ...
Neil Slater's user avatar
  • 32.1k
2 votes
Accepted

Does the state space of an MDP change in these two examples?

Whilst engineering solutions in reinforcement learning, I think it is common to discuss the concept of state space loosely, in terms of what the search space looks like for the algorithm, and what ...
Neil Slater's user avatar
  • 32.1k
2 votes

Is the described Q-table considered large?

There a couple of "rules of thumb" you might apply to decide whether a Q table is large enough that some kind of approximation would help: Does it fit into memory? Does the rate of ...
Neil Slater's user avatar
  • 32.1k
2 votes

What is the state space of digit recognition agent on a grid of 28 x 28 pixels?

State space in contextual bandits or reinforcement learning covers the input space to a policy or predictive model. The question also specifies a representation for this input, which is different in ...
Neil Slater's user avatar
  • 32.1k
1 vote
Accepted

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?

With tabular reinforcement learning (RL) methods, then catastrophic forgetting does not come into play, as it is a feature of online learning with approximators such as neural networks. Essentially ...
Neil Slater's user avatar
  • 32.1k
1 vote
Accepted

Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?

$$R_{s}=\mathbb{E}[R_{t}|S_{t}=s]$$ is the expected reward at time step $t$ given that the state at time $t$ is $s$, where $R_{t}$ and $S_t$ are random variables that represent the reward and state ...
nbro's user avatar
  • 40.6k
1 vote
Accepted

Why do terms in the computation of state space size scale exponentially?

Intuitively, I feel like if there are 30 foods, each with 2 states, then that is 60 states, no $2^{30}$. Let's try it with 3 pellets. If you are right there would be $2 \times 3 = 6$ states, if the ...
Neil Slater's user avatar
  • 32.1k
1 vote
Accepted

How DFS may expand the same state many times via different paths in an acyclic state space?

First, note that the verb to expand has a specific meaning in this context: when you expand a node/state $s$, you try each action $a_1, \dots, a_n$ available from $s$, and each of these actions $a_i$ ...
nbro's user avatar
  • 40.6k
1 vote
Accepted

How to incorporate action information in the state input of a DQN?

Drivers are not actions in this case, they are objects that are part of the state space, your state vector would look something like this \begin{equation} \mathbf{x} = [x_{o}^T, x_1^T,\ldots, x_N^T]^T ...
Brale's user avatar
  • 2,386
1 vote

Is there any inherent assumption of start and goal states in an MDP?

Is there any reason for not specifying start and goal states in MDP like in a finite automaton? In general MDPs have a start state distribution. That may be a single state, but does not have to be. ...
Neil Slater's user avatar
  • 32.1k
1 vote
Accepted

Q-learning in gridworld with random board

From your linked description of the game, we can see it has a key property when used normally in AI teaching: Partially observable: The Wumpus world is partially observable because the agent can only ...
Neil Slater's user avatar
  • 32.1k

Only top scored, non community-wiki answers of a minimum length are eligible