13 votes
Accepted

Is there a fundamental difference between an environment being stochastic and being partially observable?

I think the distinction is made more for conceptual reasons, which has practical implications, so let me review the usual definitions of a stochastic and partially observable environment. A stochastic ...
nbro's user avatar
  • 40.6k
11 votes

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally ...
Neil Slater's user avatar
  • 32.1k
8 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an ...
ComputerScientist's user avatar
8 votes

Is there a fundamental difference between an environment being stochastic and being partially observable?

A few points I'd like to add (without repeating the info already provided by nbro's answer): I think you're half-right, in that indeed we can probably always model randomness as hidden information (e....
Dennis Soemers's user avatar
  • 10.3k
6 votes

How does Q-learning work in stochastic environments?

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case? The ...
Dennis Soemers's user avatar
  • 10.3k
6 votes

Is the optimal policy always stochastic if the environment is also stochastic?

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
Adrien Forbu's user avatar
6 votes
Accepted

How do you deal with movement inertia in an environment after a step?

The simplest answer is that the inertia (or velocity, if the mass of the ball is not a variable) should be part of the observable state that the agent has access to. In RL there are usually two parts ...
Neil Slater's user avatar
  • 32.1k
5 votes
Accepted

Benchmarks for reinforcement learning in discrete MDPs

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms. MDP vs POMDP In ...
agold's user avatar
  • 375
5 votes
Accepted

How to deal with changing environment in reinforcement learning

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle? How does the agent know that it is ...
Lars's user avatar
  • 189
4 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try to figure out what exactly does an action do using such script: ...
Icyblade's user avatar
  • 141
4 votes

How to create a custom environment for reinforcement learning

This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment. Yes you will very likely need to write software to represent your environment in some ...
Neil Slater's user avatar
  • 32.1k
4 votes
Accepted

Reinforcement learning with action consisting of two discrete values

You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem ...
David's user avatar
  • 4,910
4 votes
Accepted

Why do all states appear identical under the function approximation in the Short Corridor task?

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely ...
Robby Goetschalckx's user avatar
4 votes
Accepted

How should I generate datasets for a SARSA agent when the environment is not simple?

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem There is no difference in concept, which is why tic-tac-toe and maze problems are used ...
Neil Slater's user avatar
  • 32.1k
4 votes

What exactly are partially observable environments?

You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like ...
Neil Slater's user avatar
  • 32.1k
4 votes

How do you deal with movement inertia in an environment after a step?

In RL usually a "discretized" time MDP is used in case of continuous time problems, with the idea that if we evaluate enough time the policy, we can approximate it the continuous case. This ...
Alberto's user avatar
  • 1,938
3 votes

How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki: The episode ends when you reach 0.5 position, or if 200 iterations are reached. There is a ...
DeepQZero's user avatar
  • 1,379
3 votes
Accepted

How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the ...
harwiltz's user avatar
  • 1,136
3 votes
Accepted

How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

In reinforcement learning, is the value of terminal/goal state always zero?

In reinforcement learning, is the value of terminal/goal state always zero? Yes, always for episodic problems, the value of a terminal state is always zero, from the definition. The value of a state $...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

OpenAI Gym: Multiple actions in one step

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own ...
stefanbschneider's user avatar
3 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment. "Each action is repeatedly performed for a ...
jaidmin's user avatar
  • 31
3 votes
Accepted

What exactly are partially observable environments?

First, note that the current state does not determine the next state. What determines the next state are the dynamics of the environment, which, in the context of reinforcement learning and, in ...
nbro's user avatar
  • 40.6k
3 votes
Accepted

What is the Bellman equation for V(s) in the case of a deterministic environment?

Your 2nd equation is the Bellman optimality equation (BOE) for $V$. So, to emphasise that, you could write it as follows $$ V^\color{red}{*}(s) = \max_a(R(s,a) + \gamma\sum_{s'} P(s,a,s') V^\color{red}...
nbro's user avatar
  • 40.6k
3 votes

What is the Bellman equation for V(s) in the case of a deterministic environment?

You're correct, that's the definition of the Bellman equation in the deterministic case. You can refer to the Wikipedia article of the Bellman equation where $F(x, a)$ is the reward function, with $x$ ...
Raphael Lopez Kaufman's user avatar
3 votes
Accepted

What kind of observation state would you give for that environment?

Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem ...
Neil Slater's user avatar
  • 32.1k
2 votes

Are all fully observable environments episodic?

No, not all fully observable environments are episodic. Let's take a look again at the definitions from the book: Fully Observable Environment (section 2.3.2) If an agent’s sensors give it access to ...
adamconkey's user avatar
2 votes
Accepted

How can a neural network work with continuous time?

By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the ...
Ugnes's user avatar
  • 2,023
2 votes

Can non-Markov environments also be deterministic?

Markov Environment is not about deterministic or stochastic. "Depends only on the current state and your action" does not mean you know what will happen(deterministic). We can have Markov + ...
S. Phil Kim's user avatar
2 votes
Accepted

Advice on creating a new environment using OpenAI Gym

This stackoverflow post provides a good answer. It recommends creating the environment as a new package. Note the updated link to the instructions on OpenAI. So that's what I did and it's honestly ...
Bill's user avatar
  • 141

Only top scored, non community-wiki answers of a minimum length are eligible