Skip to main content

Questions tagged [environment]

For questions related to the concept of environment in reinforcement learning and other AI sub-fields.

Filter by
Sorted by
Tagged with
16 votes
3 answers
4k views

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? Intuitively, if the environment is ...
nbro's user avatar
  • 40.9k
13 votes
2 answers
2k views

Is there a fundamental difference between an environment being stochastic and being partially observable?

In AI literature, deterministic vs stochastic and being fully-observable vs partially observable are usually considered two distinct properties of the environment. I'm confused about this because what ...
martinkunev's user avatar
10 votes
3 answers
18k views

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent? [closed]

Printing action_space for Pong-v0 gives Discrete(6) as output, i.e. $0, 1, 2, 3, 4, 5$ are actions defined in the environment as ...
cur10us's user avatar
  • 211
10 votes
1 answer
4k views

How does Q-learning work in stochastic environments?

The Q function uses the (current and future) states to determine the action that gets the highest reward. However, in a stochastic environment, the current action (at the current state) does not ...
redlum's user avatar
  • 101
7 votes
1 answer
1k views

Are all fully observable environments episodic?

According to the definition of a fully observable environment in Russell & Norvig, AIMA (2nd ed), pages 41-44, an environment is only fully observable if it requires zero memory for an agent to ...
Francis M. Bacon's user avatar
6 votes
3 answers
3k views

What exactly are partially observable environments?

I have trouble understanding the meaning of partially observable environments. Here's my doubt. According to what I understand, the state of the environment is what precisely determines the next state ...
CHANDRASEKHAR HETHA HAVYA's user avatar
6 votes
1 answer
426 views

Benchmarks for reinforcement learning in discrete MDPs

To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...
user76284's user avatar
  • 347
6 votes
1 answer
178 views

Interesting examples of discrete stochastic games

SGs are a generalization of MDPs to multiple agents. Like this previous question on MDPs, are there any interesting examples of zero-sum, discrete SGs—preferably with small state and action spaces? I'...
user76284's user avatar
  • 347
5 votes
1 answer
6k views

How to create a custom environment for reinforcement learning

I am a newbie in reinforcement learning working on a college project. The project is related to optimizing the hardware power. I am running proprietary software in Linux distribution (16.04). The goal ...
NewToCoding's user avatar
5 votes
1 answer
560 views

Why is it recommended to use a "separate test environment" when evaluating a model?

I am training an agent (stable baselines3 algorithm) on a custom environment. During training, I want to have a callback so that for every $N$ steps of the learning process, I get the current model ...
jgklsdjfgkldsfaSDF's user avatar
4 votes
2 answers
2k views

How to deal with changing environment in reinforcement learning

I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the ...
ashesofphoenix's user avatar
4 votes
2 answers
163 views

Why do all states appear identical under the function approximation in the Short Corridor task?

This is the Short Corridor problem taken from the Sutton & Barto book. Here it's written: The problem is difficult because all the states appear identical under the function approximation But ...
ZERO NULLS's user avatar
4 votes
1 answer
219 views

How should I generate datasets for a SARSA agent when the environment is not simple?

I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. ...
Shahin's user avatar
  • 153
4 votes
2 answers
612 views

How can a neural network work with continuous time?

I have an ANN model that receives an input and produces an output. The output is an action that interacts with the environment and changes the input accordingly. The network has a desired environment ...
Emad's user avatar
  • 183
4 votes
1 answer
867 views

What is the advantage of using more than one environment with the advantage actor-critic?

make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4")) envs = [make_env() for _ in range(NUM_ENVS)] Here is a code you can look at. ...
jgauth's user avatar
  • 261
4 votes
1 answer
483 views

How to assign rewards in a non-Markovian environment?

I am quite new to the Reinforcement Learning domain and I am curious about something. It seems to be the case that the majority of current research assumes Markovian environments, that is, future ...
thulungair's user avatar
4 votes
2 answers
155 views

How to represent players in a multi agent environment so each model can distinguish its own player

So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's ...
Milky's user avatar
  • 41
3 votes
2 answers
574 views

How do you deal with movement inertia in an environment after a step?

I was wondering how can we deal with movement inertia in an environment that is constantly changing? Imagine that you make a step on an environment that moves a ball. When you make the step, you make ...
CyDevos's user avatar
  • 145
3 votes
2 answers
3k views

How does an episode end in OpenAI Gym's "MountainCar-v0" environment? [closed]

I am working on OpenAI's "MountainCar-v0" environment. In this environment, each step that an agent takes returns (among other values) the variable named ...
SJa's user avatar
  • 393
3 votes
1 answer
331 views

Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
GUZ's user avatar
  • 133
3 votes
1 answer
410 views

How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

I'm training a DQN in a real environment where I do not have a natural terminal state, so I've built the episode in an artificial way (i.e. it starts in a random condition and after T steps it ends). ...
unter_983's user avatar
  • 331
3 votes
1 answer
2k views

In reinforcement learning, is the value of terminal/goal state always zero?

Let's assume we are in a $3 \times 3$ grid world with states numbered as $0,1, \dots, 8$. Suppose that the goal state is $8$, the reward of landing in the goal state is $10$, and the reward of just ...
Bhuwan Bhatt's user avatar
3 votes
1 answer
434 views

Is it really hard to learn in a stochastic environment?

I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action $a$ (But the probability to change to a not desire state is fixed, ...
Pulse9's user avatar
  • 282
3 votes
2 answers
262 views

Can non-Markov environments also be deterministic?

The definition of deterministic environment I am familiar with goes as follows: The next state of the agent depends only on the current state and the action chosen by the agent. By exclusion, ...
user9007131's user avatar
3 votes
1 answer
406 views

What should the action space for the card game Crib be?

I'm working on creating an environment for a card game, which the agent chooses to discard certain cards in the first phase of the game, and uses the remaining cards to play with. (The game is Crib if ...
Jordan Coil's user avatar
3 votes
1 answer
95 views

What are the strategies for computationally heavy environments or long-time waiting environments?

I have an environment that is computationally heavy (takes several seconds to get a reward and next state). This limits reinforcement capability, due to poor sampling of the problem. There is any ...
Daniel Wiczew's user avatar
2 votes
2 answers
1k views

What is the Bellman equation for V(s) in the case of a deterministic environment?

I am currently trying to practice reinforcement learning for an agent on a grid. The grid is deterministic. Since the grid is deterministic, to calculate the value for each grid square from the reward ...
Krellex's user avatar
  • 145
2 votes
1 answer
338 views

How does the learning rate $\alpha$ vary in stationary and non-stationary environments?

In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function: Monte Carlo method: $V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$. TD(0) method: ...
user529295's user avatar
2 votes
1 answer
828 views

What are episodic and non-episodic domains in reinforcement learning?

I was reading about the temporal difference (TD) learning and I read that: TD handles continuing, non-episodic domains Assuming that continuing means non-terminating, what does non-episodic or ...
devidduma's user avatar
  • 552
2 votes
1 answer
1k views

Is there a way to train an RL agent without any environment?

Following Deep Q-learning from Demonstrations, I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the ...
Angelo's user avatar
  • 211
2 votes
1 answer
103 views

What components of reinforcement learning influence the result the most?

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it: Formalising the agent environment (like the design of state-, ...
kitaird's user avatar
  • 115
2 votes
0 answers
171 views

What to look out for when designing an environment regarding observations?

When designing an environment, what should one look out for when designing the observation space to make the environment as easy to be learnable for an agent as possible? E.g. make sure the markov ...
kitaird's user avatar
  • 115
2 votes
0 answers
91 views

What is the appropriate way of passing a list of integers that represents the environment to a neural network's dense layer?

I'm training an RL agent using the DQN algorithm to do a specific task. The environment is represented by a list of $10$ integer numbers from $0$ to $20$. An example would be $[5, 15, 8, 8, 0, \dots]$....
mark mark's user avatar
  • 773
2 votes
0 answers
84 views

Should the RL agent be trained in an environment with real-world data or with a synthetic model?

I want to train a reinforcement learning agent in an environment with parameters (for example, the wind speed, sun irradiation, etc.) that change over time. I have recorded a limited amount of data ...
flxh's user avatar
  • 131
2 votes
1 answer
790 views

Can an RL algorithm trained in one environment be successful in a different one?

Can an RL algorithm trained in one environment be successful in a different one? For example, if I train a model to go through one labyrinth, could this model also go through a different but similar ...
GKozinski's user avatar
  • 1,270
1 vote
1 answer
115 views

What kind of observation state would you give for that environment?

I'm making a new environment where I have two sphere (one above the other) in a 2D plan. I would like some advice on what observation state I should give to my RL. Today I have given the following: ...
CyDevos's user avatar
  • 145
1 vote
1 answer
3k views

OpenAI Gym: Multiple actions in one step

I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the ...
stefanbschneider's user avatar
1 vote
1 answer
3k views

Should I make my environment with gym or gymnasium?

For context, I am looking to make my own custom Gym environment because I am more interested in trying a bunch of different architectures on this one problem than I am in seeing how a given model ...
Justin T's user avatar
  • 145
1 vote
1 answer
308 views

When is it necessary to explicitly define both the state and observation space in a custom environment?

I'm fairly new to reinforcement learning concepts, and I'm trying to implement a simple custom environment. In my custom environment, I have a scenario where I have multiple continuous state spaces, ...
AlphaBit95's user avatar
1 vote
1 answer
741 views

Advice on creating a new environment using OpenAI Gym [closed]

I'm looking for some general advice here before I dive in. I'm interested in creating a new environment for OpenAI gym to provide some slightly more challenging continuous control problems than the ...
Bill's user avatar
  • 141
1 vote
1 answer
299 views

How to normalizing various elements of the reward function?

Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
Hadar Sharvit's user avatar
1 vote
1 answer
2k views

Creating DQN Learning Agent without Gym environment for a custom project

In a project for college I created a simple turn based game, with up to 4 players that can either move or attack the opponents. The players are playing over the network, meaning the clients are ...
monkey_ball's user avatar
1 vote
1 answer
250 views

If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?

I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...
RL_NOOB's user avatar
  • 15
1 vote
1 answer
425 views

How can reinforcement learning be applied when the goal location or environment is unknown?

I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action. Take the maze problem as an example. Each ...
sue's user avatar
  • 11
1 vote
1 answer
45 views

Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters). I'm comparing my solution with some ...
Kris's user avatar
  • 171
1 vote
1 answer
325 views

Do we need multiple parallel environments to train in batches an on-policy algorithm?

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
Daniel's user avatar
  • 111
1 vote
1 answer
83 views

How to deal with the addition of a new state to the environment during training?

Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state. When it gets to that new ...
Chukwudi's user avatar
  • 369
1 vote
0 answers
111 views

Designing Reinforcement Learning Environments for Continuing Tasks (Non-Episodic) [closed]

I want to create a continuing (non-episodic) reinforcement learning environment for chiller plant operation. There are some tutorials focusing on creating environments for the episodic case, however I ...
imantha's user avatar
  • 111
1 vote
0 answers
78 views

How many parameters of a RL environment?

I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters. Given its stochasticity, I ...
Damuna Taliffato's user avatar
1 vote
0 answers
148 views

Is object-based representation of the observation space feasible?

I just started working on a DRL project from scratch. The state of each episode can be expressed as a state set $S=(S^A, S^B, S^C, S^D)$. Each subset is a feature set of a constituent component of the ...
Shahin's user avatar
  • 153