Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [state-spaces]

For questions about state spaces, in the context of reinforcement learning or other AI sub-fields.

Filter by
Sorted by
Tagged with
0 votes
1 answer
42 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
user avatar
  • 33
2 votes
1 answer
64 views

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?

I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
user avatar
  • 33
2 votes
1 answer
148 views

Is it appropriate to represent 'total failure' as an absorbing state?

My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing ...
user avatar
  • 121
0 votes
1 answer
31 views

Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?

I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
user avatar
  • 489
1 vote
1 answer
66 views

Does the state space of an MDP change in these two examples?

In the classic Atari environments, like that introduced in the original DQN paper, the state space is the set of all possible images that the Atari emulator can produce (or more generally just any RGB ...
user avatar
1 vote
1 answer
37 views

Why do terms in the computation of state space size scale exponentially?

The image below is from a Berkeley AI course pdf I found. My question is, why do the terms accounting for the ghosts and pellets come in raised to the number of units? For example, there are two ...
user avatar
3 votes
1 answer
60 views

How DFS may expand the same state many times via different paths in an acyclic state space?

I am reading the book titled Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig (4th edition) and came across this sentence about depth-first search (page 79, line 12): For ...
user avatar
4 votes
1 answer
93 views

How to approach a blackjack-like card game with the possibility of cards being counted?

Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.: You're playing against a dealer with ...
user avatar
0 votes
1 answer
75 views

How to incorporate action information in the state input of a DQN?

I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and ...
user avatar
  • 3
2 votes
1 answer
77 views

Is there any inherent assumption of start and goal states in an MDP?

MDP stands for the Markov decision process. It is a 5-length tuple used in reinforcement learning. $$MDP = (S, A, T, R, \pi)$$ $S$ stands for a set of states, also called state space. $A$ stands for a ...
user avatar
  • 3,059
0 votes
0 answers
41 views

Defining states and possible actions in Q learning

I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
user avatar
  • 1
2 votes
1 answer
746 views

What is the difference between terminal state, nonterminal states and normal states?

In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following: Each episode ends in a special state called the terminal state But the ...
user avatar
  • 489
6 votes
3 answers
829 views

What exactly are partially observable environments?

I have trouble understanding the meaning of partially observable environments. Here's my doubt. According to what I understand, the state of the environment is what precisely determines the next state ...
user avatar
1 vote
0 answers
40 views

Is there any thumb rule on the cardinality of state space in order to use the parameterized function to estimate value functions?

Value functions for a given MDP can be learned in at least two ways by experience. The first way (tabular calculation) is generally used in the case of state spaces that are small enough. The second ...
user avatar
  • 3,059
1 vote
1 answer
200 views

What is the difference between environment states and agent states in terms of Markov property?

I'm going through the David Silver RL course on YouTube. He talks about environment internal state $S^e_t$, and agent internal state $S^a_t$. We know that state $s$ is Markov if $$\mathbb{P}\{S_t=s|S_{...
user avatar
1 vote
1 answer
156 views

Q-learning in gridworld with random board

I'm trying to use Q-learning in order to solve Wumpus world environment. Wumpus world is a toy problem on 4x4 gridworld. The agent starts in entry position of the cave, looks for gold (agent can sense ...
user avatar
  • 25
3 votes
1 answer
107 views

How can I compress the states of a reinforcement learning agent?

I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like ...
user avatar
1 vote
1 answer
83 views

Compute state space from variables in Q-learning (RL)

I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state. Let's say, in my problem, there are the following variables, which I'm using to compute state: ...
user avatar
  • 25
1 vote
0 answers
55 views

What's the best way to take a list of lists as DQN input?

I have my own environment for the DQN algorithm. In my environment, the state space is represented by a list of lists, where each sublist can be of different lengths. In my case, the length of the ...
user avatar
  • 11
5 votes
2 answers
759 views

What is the relation between the context in contextual bandits and the state in reinforcement learning?

Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)? Specifically, in RL, we can use a function approximator (e.g. ...
user avatar
0 votes
0 answers
68 views

How should I model the state and action spaces for a problem where the goal is to draw a line between two points?

I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image. I have built my DQN. For now, the output layer's size of the network is $[...
user avatar
  • 323
1 vote
2 answers
158 views

Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?

I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs. To ask my question straightforwardly: is there a natural way to define the terminal state from the ...
user avatar
  • 111
2 votes
2 answers
339 views

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

Problem description: Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way: if current action ==...
user avatar
  • 23
2 votes
1 answer
82 views

How does replacing states with latent representations help RL agents?

I have seen many papers using autoencoders to replace images (states) with latent representations. Some of those methods have shown higher rewards using such techniques. However, I do not understand ...
user avatar
1 vote
0 answers
70 views

Is object-based representation of the observation space feasible?

I just started working on a DRL project from scratch. The state of each episode can be expressed as a state set $S=(S^A, S^B, S^C, S^D)$. Each subset is a feature set of a constituent component of the ...
user avatar
  • 153
4 votes
0 answers
88 views

How to update the observation probabilities in a POMDP?

How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model? The POMDP relies on observation probabilities that match an observation to a state....
user avatar
  • 65
1 vote
1 answer
46 views

Does it make sense to include constant states into reinforcement learning formulation?

Does it make sense to incorporate constant states in the Markov Decision Process and employ a reinforcement learning algorithm to solve it? For instance, for applications of personalization, I would ...
user avatar
3 votes
2 answers
289 views

What happens when the agent faces a state that never before encountered?

I have a network with nodes and links, each of them with a certain amount of resources (that can take discrete values) at the initial state. At random time steps, a service is generated, and, based on ...
user avatar
  • 155
2 votes
1 answer
76 views

Do I need to know in advance all possible number of states in Q-Learning?

In Q-learning, is it mandatory to know all possible states that can the agent may end up in? I have a network with 4 source nodes, 3 sink nodes, and 4 main links. The initial state is the status ...
user avatar
  • 155
2 votes
1 answer
347 views

What constitutes a large space state (in Q-learning)?

I know this might be specific to different problems, but does anyone know if there is any rule of thumb or references on what constitutes a large state space? I know that, according to multiple papers,...
user avatar
1 vote
0 answers
102 views

Does the order in which the features are concatenated to create the state (or observation) matter?

I'm experimenting with an RL agent that interacts with the following environment. The learning algorithm is double DQN. The neural network represents the function from state to action. It's build with ...
user avatar
  • 663
1 vote
0 answers
138 views

To solve chess with deep RL and MCTS, how should I represent the input (the state) to a neural network?

I'm wanting to build a NN that can create a policy for each possible state. I want to combine this with MCTS to eliminate randomness so when expansion occurs, I can get the probability of the move to ...
user avatar
1 vote
1 answer
76 views

What do the state features of KukaGymEnv represent? [closed]

I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv. To formulate the feature vector for the goal state, I need to know what the features of the state of the ...
user avatar
2 votes
0 answers
51 views

When to do discretization to decrease the state/action space in RL?

When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?
user avatar
  • 889
5 votes
1 answer
295 views

It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space ...
user avatar
  • 145
2 votes
0 answers
50 views

How does Hindsight Experience Replay cope with multiple goals?

What if there are multiple goals? For example, let's consider the bit-flipping environment as described in the paper HER with one small change: now, the goal is not some specific configuration, but ...
user avatar
  • 61
5 votes
3 answers
2k views

What is a high dimensional state in reinforcement learning?

In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology. Suppose my state is a high dimensional vector of length $N$, where $N$ is a ...
user avatar
18 votes
2 answers
13k views

How to define states in reinforcement learning?

I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP. What I don't understand is the process of ...
user avatar
  • 283
1 vote
1 answer
43 views

Which "assumptions" made about the state space are Russell and Norvig referring to in their book?

I am reading the cornerstone book, "Artificial Intelligence, A Modern Approach" by Stuart Russel, and Peter Norvig, and there is a passage in the book on page 98: The complexity results ...
user avatar
6 votes
1 answer
3k views

What are the state space and the state transition function in AI?

I'm studying for my AI final exam, and I'm stuck in the state space representation. I understand initial and goal states, but what I don't understand is the state space and state transition function. ...
user avatar
6 votes
1 answer
409 views

What techniques are used to make MDP discrete state space manageable?

Generating a discretized state space for an MDP (Markov Decision Process) model seems to suffer from the curse of dimensionality. Supposed my state has a few simple features: Feeling: Happy/Neutral/...
user avatar