Questions tagged [state-spaces]

For questions about state spaces, in the context of reinforcement learning or other AI sub-fields.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Devise a model/Goal Based Agent for simple Pacman Game

I have to program simple pacman game in figure below that consisting of 4*4 grid (not GUI based). Explaination The starting point of pacman is cell 0 and its goal is to consume/eat maximum food ...
0 votes
1 answer
51 views

What would the "state space" and its Python implementation be for my simulation?

Context I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. ...
0 votes
0 answers
38 views

Can action space be part of the state space?

I'm working on a project where I have access to position coordinates and velocity components of multiple agents in an environment. Assuming that one agent is controllable while others are not ...
1 vote
1 answer
76 views

Is the described Q-table considered large?

I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
0 votes
1 answer
92 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
  • 33
2 votes
1 answer
104 views

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?

I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
  • 33
2 votes
1 answer
174 views

Is it appropriate to represent 'total failure' as an absorbing state?

My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing ...
  • 121
0 votes
1 answer
40 views

Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?

I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
  • 509
1 vote
1 answer
140 views

Does the state space of an MDP change in these two examples?

In the classic Atari environments, like that introduced in the original DQN paper, the state space is the set of all possible images that the Atari emulator can produce (or more generally just any RGB ...
1 vote
1 answer
82 views

Why do terms in the computation of state space size scale exponentially?

The image below is from a Berkeley AI course pdf I found. My question is, why do the terms accounting for the ghosts and pellets come in raised to the number of units? For example, there are two ...
3 votes
1 answer
378 views

How DFS may expand the same state many times via different paths in an acyclic state space?

I am reading the book titled Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig (4th edition) and came across this sentence about depth-first search (page 79, line 12): For ...
4 votes
1 answer
181 views

How to approach a blackjack-like card game with the possibility of cards being counted?

Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.: You're playing against a dealer with ...
0 votes
1 answer
120 views

How to incorporate action information in the state input of a DQN?

I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and ...
  • 3
2 votes
1 answer
187 views

Is there any inherent assumption of start and goal states in an MDP?

MDP stands for the Markov decision process. It is a 5-length tuple used in reinforcement learning. $$MDP = (S, A, T, R, \pi)$$ $S$ stands for a set of states, also called state space. $A$ stands for a ...
  • 3,451
0 votes
0 answers
50 views

Defining states and possible actions in Q learning

I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
  • 1
3 votes
1 answer
2k views

What is the difference between terminal state, nonterminal states and normal states?

In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following: Each episode ends in a special state called the terminal state But the ...
  • 509
6 votes
3 answers
2k views

What exactly are partially observable environments?

I have trouble understanding the meaning of partially observable environments. Here's my doubt. According to what I understand, the state of the environment is what precisely determines the next state ...
1 vote
0 answers
54 views

Is there any thumb rule on the cardinality of state space in order to use the parameterized function to estimate value functions?

Value functions for a given MDP can be learned in at least two ways by experience. The first way (tabular calculation) is generally used in the case of state spaces that are small enough. The second ...
  • 3,451
1 vote
1 answer
513 views

What is the difference between environment states and agent states in terms of Markov property?

I'm going through the David Silver RL course on YouTube. He talks about environment internal state $S^e_t$, and agent internal state $S^a_t$. We know that state $s$ is Markov if $$\mathbb{P}\{S_t=s|S_{...
1 vote
1 answer
246 views

Q-learning in gridworld with random board

I'm trying to use Q-learning in order to solve Wumpus world environment. Wumpus world is a toy problem on 4x4 gridworld. The agent starts in entry position of the cave, looks for gold (agent can sense ...
  • 25
3 votes
1 answer
181 views

How can I compress the states of a reinforcement learning agent?

I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like ...
1 vote
1 answer
185 views

Compute state space from variables in Q-learning (RL)

I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state. Let's say, in my problem, there are the following variables, which I'm using to compute state: ...
  • 25
1 vote
0 answers
57 views

What's the best way to take a list of lists as DQN input?

I have my own environment for the DQN algorithm. In my environment, the state space is represented by a list of lists, where each sublist can be of different lengths. In my case, the length of the ...
  • 11
6 votes
2 answers
2k views

What is the relation between the context in contextual bandits and the state in reinforcement learning?

Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)? Specifically, in RL, we can use a function approximator (e.g. ...
0 votes
0 answers
144 views

How should I model the state and action spaces for a problem where the goal is to draw a line between two points?

I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image. I have built my DQN. For now, the output layer's size of the network is $[...
  • 323
1 vote
2 answers
298 views

Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?

I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs. To ask my question straightforwardly: is there a natural way to define the terminal state from the ...
  • 111
2 votes
2 answers
854 views

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

Problem description: Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way: if current action ==...
  • 23
2 votes
1 answer
99 views

How does replacing states with latent representations help RL agents?

I have seen many papers using autoencoders to replace images (states) with latent representations. Some of those methods have shown higher rewards using such techniques. However, I do not understand ...
1 vote
0 answers
84 views

Is object-based representation of the observation space feasible?

I just started working on a DRL project from scratch. The state of each episode can be expressed as a state set $S=(S^A, S^B, S^C, S^D)$. Each subset is a feature set of a constituent component of the ...
  • 153
4 votes
0 answers
110 views

How to update the observation probabilities in a POMDP?

How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model? The POMDP relies on observation probabilities that match an observation to a state....
  • 85
1 vote
1 answer
64 views

Does it make sense to include constant states into reinforcement learning formulation?

Does it make sense to incorporate constant states in the Markov Decision Process and employ a reinforcement learning algorithm to solve it? For instance, for applications of personalization, I would ...
3 votes
2 answers
306 views

What happens when the agent faces a state that never before encountered?

I have a network with nodes and links, each of them with a certain amount of resources (that can take discrete values) at the initial state. At random time steps, a service is generated, and, based on ...
  • 155
2 votes
1 answer
148 views

Do I need to know in advance all possible number of states in Q-Learning?

In Q-learning, is it mandatory to know all possible states that can the agent may end up in? I have a network with 4 source nodes, 3 sink nodes, and 4 main links. The initial state is the status ...
  • 155
2 votes
1 answer
954 views

What constitutes a large space state (in Q-learning)?

I know this might be specific to different problems, but does anyone know if there is any rule of thumb or references on what constitutes a large state space? I know that, according to multiple papers,...
1 vote
0 answers
160 views

Does the order in which the features are concatenated to create the state (or observation) matter?

I'm experimenting with an RL agent that interacts with the following environment. The learning algorithm is double DQN. The neural network represents the function from state to action. It's build with ...
  • 703
1 vote
0 answers
210 views

To solve chess with deep RL and MCTS, how should I represent the input (the state) to a neural network?

I'm wanting to build a NN that can create a policy for each possible state. I want to combine this with MCTS to eliminate randomness so when expansion occurs, I can get the probability of the move to ...
1 vote
1 answer
84 views

What do the state features of KukaGymEnv represent? [closed]

I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv. To formulate the feature vector for the goal state, I need to know what the features of the state of the ...
2 votes
0 answers
71 views

When to do discretization to decrease the state/action space in RL?

When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?
  • 909
5 votes
1 answer
335 views

It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space ...
  • 145
2 votes
0 answers
55 views

How does Hindsight Experience Replay cope with multiple goals?

What if there are multiple goals? For example, let's consider the bit-flipping environment as described in the paper HER with one small change: now, the goal is not some specific configuration, but ...
  • 61
6 votes
3 answers
2k views

What is a high dimensional state in reinforcement learning?

In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology. Suppose my state is a high dimensional vector of length $N$, where $N$ is a ...
19 votes
2 answers
15k views

How to define states in reinforcement learning?

I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP. What I don't understand is the process of ...
  • 293
1 vote
1 answer
52 views

Which "assumptions" made about the state space are Russell and Norvig referring to in their book?

I am reading the cornerstone book, "Artificial Intelligence, A Modern Approach" by Stuart Russel, and Peter Norvig, and there is a passage in the book on page 98: The complexity results ...
6 votes
1 answer
3k views

What are the state space and the state transition function in AI?

I'm studying for my AI final exam, and I'm stuck in the state space representation. I understand initial and goal states, but what I don't understand is the state space and state transition function. ...
6 votes
1 answer
505 views

What techniques are used to make MDP discrete state space manageable?

Generating a discretized state space for an MDP (Markov Decision Process) model seems to suffer from the curse of dimensionality. Supposed my state has a few simple features: Feeling: Happy/Neutral/...