12
votes
Accepted
Is there a fundamental difference between an environment being stochastic and being partially observable?
I think the distinction is made more for conceptual reasons, which has practical implications, so let me review the usual definitions of a stochastic and partially observable environment.
A stochastic ...
- 39.1k
11
votes
Is the optimal policy always stochastic if the environment is also stochastic?
Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic?
No.
An optimal policy is generally ...
- 29k
8
votes
What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?
You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub.
In particular, 0 means no action, 1 means fire, which is why they don't have an ...
8
votes
Is there a fundamental difference between an environment being stochastic and being partially observable?
A few points I'd like to add (without repeating the info already provided by nbro's answer):
I think you're half-right, in that indeed we can probably always model randomness as hidden information (e....
- 9,944
6
votes
Is the optimal policy always stochastic if the environment is also stochastic?
I would say no.
For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
- 251
6
votes
Accepted
How do you deal with movement inertia in an environment after a step?
The simplest answer is that the inertia (or velocity, if the mass of the ball is not a variable) should be part of the observable state that the agent has access to. In RL there are usually two parts ...
- 29k
5
votes
Accepted
Benchmarks for reinforcement learning in discrete MDPs
Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms.
MDP vs POMDP
In ...
- 365
5
votes
How does Q-learning work in stochastic environments?
How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?
The ...
- 9,944
4
votes
What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?
You can try to figure out what exactly does an action do using such script:
...
- 141
4
votes
How to create a custom environment for reinforcement learning
This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment.
Yes you will very likely need to write software to represent your environment in some ...
- 29k
4
votes
Accepted
Reinforcement learning with action consisting of two discrete values
You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem ...
- 4,615
4
votes
Accepted
Why do all states appear identical under the function approximation in the Short Corridor task?
You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely ...
4
votes
Accepted
How should I generate datasets for a SARSA agent when the environment is not simple?
I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem
There is no difference in concept, which is why tic-tac-toe and maze problems are used ...
- 29k
4
votes
What exactly are partially observable environments?
You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like ...
- 29k
4
votes
Accepted
How to deal with changing environment in reinforcement learning
I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle?
How does the agent know that it is ...
- 179
4
votes
How do you deal with movement inertia in an environment after a step?
In RL usually a "discretized" time MDP is used in case of continuous time problems, with the idea that if we evaluate enough time the policy, we can approximate it the continuous case. This ...
- 911
3
votes
How does an episode end in OpenAI Gym's "MountainCar-v0" environment?
To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki:
The episode ends when you reach 0.5 position, or if 200 iterations are reached.
There is a ...
- 1,192
3
votes
Accepted
How does an episode end in OpenAI Gym's "MountainCar-v0" environment?
The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the ...
- 1,106
3
votes
Accepted
How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?
If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an ...
- 29k
3
votes
Accepted
In reinforcement learning, is the value of terminal/goal state always zero?
In reinforcement learning, is the value of terminal/goal state always zero?
Yes, always for episodic problems, the value of a terminal state is always zero, from the definition.
The value of a state $...
- 29k
3
votes
Accepted
OpenAI Gym: Multiple actions in one step
What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own ...
- 183
3
votes
What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?
There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment.
"Each action is repeatedly performed for a ...
- 31
3
votes
Accepted
What exactly are partially observable environments?
First, note that the current state does not determine the next state. What determines the next state are the dynamics of the environment, which, in the context of reinforcement learning and, in ...
- 39.1k
3
votes
Accepted
What is the Bellman equation for V(s) in the case of a deterministic environment?
Your 2nd equation is the Bellman optimality equation (BOE) for $V$. So, to emphasise that, you could write it as follows
$$
V^\color{red}{*}(s)
=
\max_a(R(s,a) + \gamma\sum_{s'} P(s,a,s') V^\color{red}...
- 39.1k
3
votes
What is the Bellman equation for V(s) in the case of a deterministic environment?
You're correct, that's the definition of the Bellman equation in the deterministic case.
You can refer to the Wikipedia article of the Bellman equation where $F(x, a)$ is the reward function, with $x$ ...
3
votes
Accepted
What kind of observation state would you give for that environment?
Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem ...
- 29k
2
votes
Are all fully observable environments episodic?
No, not all fully observable environments are episodic. Let's take a look again at the definitions from the book:
Fully Observable Environment (section 2.3.2)
If an agent’s sensors give it access to ...
- 271
2
votes
Accepted
How can a neural network work with continuous time?
By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the ...
- 2,023
2
votes
Can non-Markov environments also be deterministic?
Markov Environment is not about deterministic or stochastic. "Depends only on the current state and your action" does not mean you know what will happen(deterministic).
We can have Markov + ...
- 151
2
votes
Accepted
Advice on creating a new environment using OpenAI Gym
This stackoverflow post provides a good answer. It recommends creating the environment as a new package.
Note the updated link to the instructions on OpenAI.
So that's what I did and it's honestly ...
- 141
Only top scored, non community-wiki answers of a minimum length are eligible