Questions tagged [state-spaces]
For questions about state spaces, in the context of reinforcement learning or other AI sub-fields.
49
questions
0
votes
0
answers
18
views
Can I use Prioritized Learning when the transition probabilities are given?
Can I use Prioritized Learning when the transition probabilities are given?
Also, as I can understand, Prioritized Sweeping is suitable when a state space is large. Can I use it when the state space ...
0
votes
0
answers
14
views
Requesting resources on causal networks for 2D strategy game
I am requesting research, articles, abstracts or interesting opinions that will help me create a complex causal neural network. There are many detailed resources on causal discovery, image recognition,...
1
vote
1
answer
99
views
What kind of observation state would you give for that environment?
I'm making a new environment where I have two sphere (one above the other) in a 2D plan.
I would like some advice on what observation state I should give to my RL.
Today I have given the following:
...
- 145
1
vote
1
answer
117
views
Implementing an RL agent on a variable action space
I've been trying to design and implement a DQN agent as an AI opponent option for a math game I've been working on.
The game is a turn-based grid game, where players alternate making moves and ...
- 11
1
vote
1
answer
188
views
Variable observation space at each episode
I have an enviroment with continuous actions and state variables. Every time I reset my env, between 2 and 5 balls spawn randomly in a box of 100x100 size. One of those balls (the red one) will ...
0
votes
1
answer
76
views
What would the "state space" and its Python implementation be for my simulation?
Context
I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. ...
- 547
0
votes
0
answers
152
views
Can action space be part of the state space?
I'm working on a project where I have access to position coordinates and velocity components of multiple agents in an environment. Assuming that one agent is controllable while others are not ...
- 133
1
vote
1
answer
251
views
Is the described Q-table considered large?
I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
0
votes
1
answer
185
views
How to manage impossible actions? [closed]
I am using Q-learning in julia language.
Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
- 33
2
votes
1
answer
247
views
Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?
I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
- 33
2
votes
1
answer
207
views
Is it appropriate to represent 'total failure' as an absorbing state?
My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing ...
- 121
0
votes
1
answer
53
views
Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?
I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
- 573
1
vote
1
answer
211
views
Does the state space of an MDP change in these two examples?
In the classic Atari environments, like that introduced in the original DQN paper, the state space is the set of all possible images that the Atari emulator can produce (or more generally just any RGB ...
- 4,685
1
vote
1
answer
137
views
Why do terms in the computation of state space size scale exponentially?
The image below is from a Berkeley AI course pdf I found.
My question is, why do the terms accounting for the ghosts and pellets come in raised to the number of units?
For example, there are two ...
3
votes
1
answer
622
views
How DFS may expand the same state many times via different paths in an acyclic state space?
I am reading the book titled Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig (4th edition) and came across this sentence about depth-first search (page 79, line 12):
For ...
- 123
4
votes
1
answer
234
views
How to approach a blackjack-like card game with the possibility of cards being counted?
Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.:
You're playing against a dealer with ...
- 41
0
votes
1
answer
198
views
How to incorporate action information in the state input of a DQN?
I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and ...
- 3
2
votes
1
answer
386
views
Is there any inherent assumption of start and goal states in an MDP?
MDP stands for the Markov decision process. It is a 5-length tuple used in reinforcement learning.
$$MDP = (S, A, T, R, \pi)$$
$S$ stands for a set of states, also called state space.
$A$ stands for a ...
- 3,622
0
votes
0
answers
69
views
Defining states and possible actions in Q learning
I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
- 101
3
votes
1
answer
3k
views
What is the difference between terminal state, nonterminal states and normal states?
In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following:
Each episode ends in a special state called the terminal state
But the ...
- 573
6
votes
3
answers
3k
views
What exactly are partially observable environments?
I have trouble understanding the meaning of partially observable environments. Here's my doubt.
According to what I understand, the state of the environment is what precisely determines the next state ...
1
vote
0
answers
73
views
Is there any thumb rule on the cardinality of state space in order to use the parameterized function to estimate value functions?
Value functions for a given MDP can be learned in at least two ways by experience.
The first way (tabular calculation) is generally used in the case of state spaces that are small enough.
The second ...
- 3,622
1
vote
1
answer
818
views
What is the difference between environment states and agent states in terms of Markov property?
I'm going through the David Silver RL course on YouTube. He talks about environment internal state $S^e_t$, and agent internal state $S^a_t$.
We know that state $s$ is Markov if
$$\mathbb{P}\{S_t=s|S_{...
1
vote
1
answer
354
views
Q-learning in gridworld with random board
I'm trying to use Q-learning in order to solve Wumpus world environment.
Wumpus world is a toy problem on 4x4 gridworld. The agent starts in entry position of the cave, looks for gold (agent can sense ...
- 25
4
votes
1
answer
265
views
How can I compress the states of a reinforcement learning agent?
I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like ...
- 123
1
vote
1
answer
300
views
Compute state space from variables in Q-learning (RL)
I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state.
Let's say, in my problem, there are the following variables, which I'm using to compute state:
...
- 25
1
vote
0
answers
67
views
What's the best way to take a list of lists as DQN input?
I have my own environment for the DQN algorithm. In my environment, the state space is represented by a list of lists, where each sublist can be of different lengths. In my case, the length of the ...
- 11
8
votes
2
answers
3k
views
What is the relation between the context in contextual bandits and the state in reinforcement learning?
Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)?
Specifically, in RL, we can use a function approximator (e.g. ...
- 183
0
votes
0
answers
169
views
How should I model the state and action spaces for a problem where the goal is to draw a line between two points?
I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image.
I have built my DQN. For now, the output layer's size of the network is $[...
- 323
1
vote
2
answers
528
views
Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?
I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.
To ask my question straightforwardly: is there a natural way to define the terminal state from the ...
- 111
3
votes
2
answers
2k
views
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
Problem description:
Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:
if current action ==...
- 33
2
votes
1
answer
125
views
How does replacing states with latent representations help RL agents?
I have seen many papers using autoencoders to replace images (states) with latent representations. Some of those methods have shown higher rewards using such techniques. However, I do not understand ...
- 618
1
vote
0
answers
140
views
Is object-based representation of the observation space feasible?
I just started working on a DRL project from scratch. The state of each episode can be expressed as a state set $S=(S^A, S^B, S^C, S^D)$. Each subset is a feature set of a constituent component of the ...
- 153
4
votes
0
answers
137
views
How to update the observation probabilities in a POMDP?
How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model?
The POMDP relies on observation probabilities that match an observation to a state....
- 85
1
vote
1
answer
86
views
Does it make sense to include constant states into reinforcement learning formulation?
Does it make sense to incorporate constant states in the Markov Decision Process and employ a reinforcement learning algorithm to solve it?
For instance, for applications of personalization, I would ...
- 341
3
votes
2
answers
365
views
What happens when the agent faces a state that never before encountered?
I have a network with nodes and links, each of them with a certain amount of resources (that can take discrete values) at the initial state. At random time steps, a service is generated, and, based on ...
- 155
2
votes
1
answer
266
views
Do I need to know in advance all possible number of states in Q-Learning?
In Q-learning, is it mandatory to know all possible states that can the agent may end up in?
I have a network with 4 source nodes, 3 sink nodes, and 4 main links. The initial state is the status ...
- 155
3
votes
1
answer
2k
views
What constitutes a large space state (in Q-learning)?
I know this might be specific to different problems, but does anyone know if there is any rule of thumb or references on what constitutes a large state space?
I know that, according to multiple papers,...
- 31
1
vote
0
answers
268
views
Does the order in which the features are concatenated to create the state (or observation) matter?
I'm experimenting with an RL agent that interacts with the following environment. The learning algorithm is double DQN. The neural network represents the function from state to action. It's build with ...
- 763
1
vote
0
answers
280
views
To solve chess with deep RL and MCTS, how should I represent the input (the state) to a neural network?
I'm wanting to build a NN that can create a policy for each possible state. I want to combine this with MCTS to eliminate randomness so when expansion occurs, I can get the probability of the move to ...
1
vote
1
answer
91
views
What do the state features of KukaGymEnv represent? [closed]
I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv.
To formulate the feature vector for the goal state, I need to know what the features of the state of the ...
- 125
2
votes
0
answers
82
views
When to do discretization to decrease the state/action space in RL?
When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?
- 919
5
votes
1
answer
367
views
It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?
I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space ...
- 145
2
votes
0
answers
58
views
How does Hindsight Experience Replay cope with multiple goals?
What if there are multiple goals? For example, let's consider the bit-flipping environment as described in the paper HER with one small change: now, the goal is not some specific configuration, but ...
- 61
6
votes
3
answers
3k
views
What is a high dimensional state in reinforcement learning?
In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology.
Suppose my state is a high dimensional vector of length $N$, where $N$ is a ...
- 163
21
votes
2
answers
16k
views
How to define states in reinforcement learning?
I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP.
What I don't understand is the process of ...
- 313
1
vote
1
answer
73
views
Which "assumptions" made about the state space are Russell and Norvig referring to in their book?
I am reading the cornerstone book, "Artificial Intelligence, A Modern Approach" by Stuart Russel, and Peter Norvig, and there is a passage in the book on page 98:
The complexity results ...
- 287
6
votes
1
answer
4k
views
What are the state space and the state transition function in AI?
I'm studying for my AI final exam, and I'm stuck in the state space representation. I understand initial and goal states, but what I don't understand is the state space and state transition function. ...
- 63
6
votes
1
answer
615
views
What techniques are used to make MDP discrete state space manageable?
Generating a discretized state space for an MDP (Markov Decision Process) model seems to suffer from the curse of dimensionality.
Supposed my state has a few simple features:
Feeling: Happy/Neutral/...
- 263