Skip to main content

Questions tagged [environment]

For questions related to the concept of environment in reinforcement learning and other AI sub-fields.

Filter by
Sorted by
Tagged with
1 vote
1 answer
115 views

What kind of observation state would you give for that environment?

I'm making a new environment where I have two sphere (one above the other) in a 2D plan. I would like some advice on what observation state I should give to my RL. Today I have given the following: ...
CyDevos's user avatar
  • 145
3 votes
2 answers
574 views

How do you deal with movement inertia in an environment after a step?

I was wondering how can we deal with movement inertia in an environment that is constantly changing? Imagine that you make a step on an environment that moves a ball. When you make the step, you make ...
CyDevos's user avatar
  • 145
1 vote
1 answer
3k views

Should I make my environment with gym or gymnasium?

For context, I am looking to make my own custom Gym environment because I am more interested in trying a bunch of different architectures on this one problem than I am in seeing how a given model ...
Justin T's user avatar
  • 145
0 votes
0 answers
19 views

When should grayscale processing be applied to image inputs in visual reinforcement learning environments?

I am currently working with visual environments in Reinforcement Learning (RL) and have noticed differing practices regarding preprocessing of image inputs. Specifically, in the Atari environment, a ...
XiaoBanni's user avatar
1 vote
0 answers
111 views

Designing Reinforcement Learning Environments for Continuing Tasks (Non-Episodic) [closed]

I want to create a continuing (non-episodic) reinforcement learning environment for chiller plant operation. There are some tutorials focusing on creating environments for the episodic case, however I ...
imantha's user avatar
  • 111
0 votes
1 answer
39 views

Are platformer games, with the camera centred on the character, examples of egocentric vision?

An example may be CoinRun, where the character sprite is always centred in the camera view, while the environment moves as a result of player input. To me, this sounds like egocentric vision, although ...
thesofakillers's user avatar
1 vote
1 answer
308 views

When is it necessary to explicitly define both the state and observation space in a custom environment?

I'm fairly new to reinforcement learning concepts, and I'm trying to implement a simple custom environment. In my custom environment, I have a scenario where I have multiple continuous state spaces, ...
AlphaBit95's user avatar
0 votes
1 answer
155 views

Why is the sliding puzzle problem episodic?

Why is the sliding puzzle problem episodic and not sequential? From what I understand, an environment is episodic if each episode is independent and doesn't affect past or future episodes. The actions ...
numq's user avatar
  • 1
5 votes
1 answer
560 views

Why is it recommended to use a "separate test environment" when evaluating a model?

I am training an agent (stable baselines3 algorithm) on a custom environment. During training, I want to have a callback so that for every $N$ steps of the learning process, I get the current model ...
jgklsdjfgkldsfaSDF's user avatar
1 vote
1 answer
299 views

How to normalizing various elements of the reward function?

Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
Hadar Sharvit's user avatar
1 vote
0 answers
78 views

How many parameters of a RL environment?

I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters. Given its stochasticity, I ...
Damuna Taliffato's user avatar
2 votes
2 answers
1k views

What is the Bellman equation for V(s) in the case of a deterministic environment?

I am currently trying to practice reinforcement learning for an agent on a grid. The grid is deterministic. Since the grid is deterministic, to calculate the value for each grid square from the reward ...
Krellex's user avatar
  • 145
4 votes
2 answers
2k views

How to deal with changing environment in reinforcement learning

I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the ...
ashesofphoenix's user avatar
2 votes
0 answers
171 views

What to look out for when designing an environment regarding observations?

When designing an environment, what should one look out for when designing the observation space to make the environment as easy to be learnable for an agent as possible? E.g. make sure the markov ...
kitaird's user avatar
  • 115
0 votes
0 answers
99 views

Which algorithms work in a non-stationary stochastic environment?

Currently, I am reading into the Multi-Armed-Bandit problem and found the special case of non-stationary (environment and its attributes, like the reward distribution, change over time) stochastic ...
paperplan3's user avatar
13 votes
2 answers
2k views

Is there a fundamental difference between an environment being stochastic and being partially observable?

In AI literature, deterministic vs stochastic and being fully-observable vs partially observable are usually considered two distinct properties of the environment. I'm confused about this because what ...
martinkunev's user avatar
2 votes
1 answer
103 views

What components of reinforcement learning influence the result the most?

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it: Formalising the agent environment (like the design of state-, ...
kitaird's user avatar
  • 115
3 votes
1 answer
432 views

Is it really hard to learn in a stochastic environment?

I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action $a$ (But the probability to change to a not desire state is fixed, ...
Pulse9's user avatar
  • 282
2 votes
1 answer
338 views

How does the learning rate $\alpha$ vary in stationary and non-stationary environments?

In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function: Monte Carlo method: $V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$. TD(0) method: ...
user529295's user avatar
1 vote
1 answer
2k views

Creating DQN Learning Agent without Gym environment for a custom project

In a project for college I created a simple turn based game, with up to 4 players that can either move or attack the opponents. The players are playing over the network, meaning the clients are ...
monkey_ball's user avatar
6 votes
3 answers
3k views

What exactly are partially observable environments?

I have trouble understanding the meaning of partially observable environments. Here's my doubt. According to what I understand, the state of the environment is what precisely determines the next state ...
CHANDRASEKHAR HETHA HAVYA's user avatar
1 vote
0 answers
148 views

Is object-based representation of the observation space feasible?

I just started working on a DRL project from scratch. The state of each episode can be expressed as a state set $S=(S^A, S^B, S^C, S^D)$. Each subset is a feature set of a constituent component of the ...
Shahin's user avatar
  • 153
4 votes
1 answer
219 views

How should I generate datasets for a SARSA agent when the environment is not simple?

I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. ...
Shahin's user avatar
  • 153
0 votes
1 answer
249 views

How to use and update a shared/global policy between Reinforcement Learning Agents

I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update ...
RL_NOOB's user avatar
  • 15
1 vote
1 answer
250 views

If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?

I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...
RL_NOOB's user avatar
  • 15
2 votes
0 answers
91 views

What is the appropriate way of passing a list of integers that represents the environment to a neural network's dense layer?

I'm training an RL agent using the DQN algorithm to do a specific task. The environment is represented by a list of $10$ integer numbers from $0$ to $20$. An example would be $[5, 15, 8, 8, 0, \dots]$....
mark mark's user avatar
  • 773
1 vote
1 answer
425 views

How can reinforcement learning be applied when the goal location or environment is unknown?

I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action. Take the maze problem as an example. Each ...
sue's user avatar
  • 11
4 votes
2 answers
163 views

Why do all states appear identical under the function approximation in the Short Corridor task?

This is the Short Corridor problem taken from the Sutton & Barto book. Here it's written: The problem is difficult because all the states appear identical under the function approximation But ...
ZERO NULLS's user avatar
0 votes
1 answer
72 views

How to let the agent choose how to populate a state space matrix in RL (using python)

I have an agent (drone) that has to allocate subchannels for different types of User Equipment. I have represented the subchannel allocation with a 2-dimentional binary matrix, that is initialized to ...
Ness's user avatar
  • 206
3 votes
1 answer
95 views

What are the strategies for computationally heavy environments or long-time waiting environments?

I have an environment that is computationally heavy (takes several seconds to get a reward and next state). This limits reinforcement capability, due to poor sampling of the problem. There is any ...
Daniel Wiczew's user avatar
0 votes
1 answer
962 views

How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?

I'm working in A2C and I have an environment where there is increasing or decreasing in the number of agents. The action space in the environment will not change but the state will change when new ...
I_Al-thamary's user avatar
3 votes
1 answer
410 views

How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

I'm training a DQN in a real environment where I do not have a natural terminal state, so I've built the episode in an artificial way (i.e. it starts in a random condition and after T steps it ends). ...
unter_983's user avatar
  • 331
0 votes
1 answer
2k views

Need some reviews in PEAS descriptions

Here is the Question: Describe the PEAS descriptions for the following agents: a) A grocery store scanner that digitally scans a fruit or vegetable and identifies it. b) A GPS system for an automobile....
Fahad's user avatar
  • 1
0 votes
0 answers
121 views

Strange behavior of Q-learning agent after being trained

I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment. Environment: A ...
SJa's user avatar
  • 393
3 votes
1 answer
331 views

Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
GUZ's user avatar
  • 133
1 vote
1 answer
45 views

Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters). I'm comparing my solution with some ...
Kris's user avatar
  • 171
3 votes
2 answers
3k views

How does an episode end in OpenAI Gym's "MountainCar-v0" environment? [closed]

I am working on OpenAI's "MountainCar-v0" environment. In this environment, each step that an agent takes returns (among other values) the variable named ...
SJa's user avatar
  • 393
1 vote
1 answer
325 views

Do we need multiple parallel environments to train in batches an on-policy algorithm?

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
Daniel's user avatar
  • 111
0 votes
2 answers
65 views

Should I build an environment from scratch myself or it is not always needed?

I am inspired by the paper Neural Architecture Search with Reinforcement Learning to use reinforcement learning for optimizing a child network (learner). My meta-learner (controller or parent network) ...
samsambakster's user avatar
1 vote
1 answer
83 views

How to deal with the addition of a new state to the environment during training?

Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state. When it gets to that new ...
Chukwudi's user avatar
  • 369
3 votes
1 answer
406 views

What should the action space for the card game Crib be?

I'm working on creating an environment for a card game, which the agent chooses to discard certain cards in the first phase of the game, and uses the remaining cards to play with. (The game is Crib if ...
Jordan Coil's user avatar
1 vote
1 answer
231 views

Once the environments are vectorized, how do I have to gather immediate experiences for the agent?

My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the ...
jgauth's user avatar
  • 261
1 vote
1 answer
3k views

OpenAI Gym: Multiple actions in one step

I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the ...
stefanbschneider's user avatar
1 vote
1 answer
531 views

How do you know if an agent has learnt its environment in reinforcement learning?

I'm new to reinforcement learning and trying to understand it. If you train an agent using a reinforcement learning algorithm (discrete or continuous) on an environment (real or simulated), then how ...
Cristian M's user avatar
4 votes
1 answer
867 views

What is the advantage of using more than one environment with the advantage actor-critic?

make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4")) envs = [make_env() for _ in range(NUM_ENVS)] Here is a code you can look at. ...
jgauth's user avatar
  • 261
1 vote
0 answers
34 views

What if the rewards induced by an environment are related to the policy too?

Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
ASA's user avatar
  • 151
2 votes
1 answer
828 views

What are episodic and non-episodic domains in reinforcement learning?

I was reading about the temporal difference (TD) learning and I read that: TD handles continuing, non-episodic domains Assuming that continuing means non-terminating, what does non-episodic or ...
devidduma's user avatar
  • 552
1 vote
0 answers
56 views

Why does this tutorial on reinforced learning not check whether the environment is 'game over' during training?

I am following the tutorial Train a Deep Q Network with TF-Agents. It uses the hello world environment of reinforced learning: cart pole. At the end, the agent is ...
Mike de Klerk's user avatar
2 votes
0 answers
84 views

Should the RL agent be trained in an environment with real-world data or with a synthetic model?

I want to train a reinforcement learning agent in an environment with parameters (for example, the wind speed, sun irradiation, etc.) that change over time. I have recorded a limited amount of data ...
flxh's user avatar
  • 131
1 vote
1 answer
240 views

How should I design the action space of an agent that needs to choose a 2d point and then shoot a cannonball?

I'm building a game environment (see the picture below) where an agent should position the mouse on the screen (see the coordinates on the upper right corner) and then click to shoot a cannonball. If ...
Voß's user avatar
  • 99