12 votes
Accepted

Is there a fundamental difference between an environment being stochastic and being partially observable?

I think the distinction is made more for conceptual reasons, which has practical implications, so let me review the usual definitions of a stochastic and partially observable environment. A stochastic ...
user avatar
  • 33.8k
9 votes

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally ...
user avatar
  • 23.8k
8 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an ...
user avatar
7 votes

Is there a fundamental difference between an environment being stochastic and being partially observable?

A few points I'd like to add (without repeating the info already provided by nbro's answer): I think you're half-right, in that indeed we can probably always model randomness as hidden information (e....
user avatar
  • 9,379
6 votes

Is the optimal policy always stochastic if the environment is also stochastic?

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
user avatar
5 votes
Accepted

Benchmarks for reinforcement learning in discrete MDPs

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms. MDP vs POMDP In ...
user avatar
  • 365
5 votes

How does Q-learning work in stochastic environments?

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case? The ...
user avatar
  • 9,379
4 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try to figure out what exactly does an action do using such script: ...
user avatar
  • 141
4 votes

How to create a custom environment for reinforcement learning

This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment. Yes you will very likely need to write software to represent your environment in some ...
user avatar
  • 23.8k
4 votes
Accepted

How should I generate datasets for a SARSA agent when the environment is not simple?

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem There is no difference in concept, which is why tic-tac-toe and maze problems are used ...
user avatar
  • 23.8k
4 votes

What exactly are partially observable environments?

You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like ...
user avatar
  • 23.8k
4 votes
Accepted

How to deal with changing environment in reinforcement learning

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle? How does the agent know that it is ...
user avatar
  • 179
3 votes

How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki: The episode ends when you reach 0.5 position, or if 200 iterations are reached. There is a ...
user avatar
  • 928
3 votes
Accepted

How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the ...
user avatar
  • 961
3 votes
Accepted

OpenAI Gym: Multiple actions in one step

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own ...
user avatar
  • 183
3 votes
Accepted

Reinforcement learning with action consisting of two discrete values

You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem ...
user avatar
3 votes
Accepted

How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an ...
user avatar
  • 23.8k
3 votes
Accepted

Why do all states appear identical under the function approximation in the Short Corridor task?

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely ...
user avatar
3 votes
Accepted

In reinforcement learning, is the value of terminal/goal state always zero?

In reinforcement learning, is the value of terminal/goal state always zero? Yes, always for episodic problems, the value of a terminal state is always zero, from the definition. The value of a state $...
user avatar
  • 23.8k
3 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment. "Each action is repeatedly performed for a ...
user avatar
  • 31
3 votes
Accepted

What exactly are partially observable environments?

First, note that the current state does not determine the next state. What determines the next state are the dynamics of the environment, which, in the context of reinforcement learning and, in ...
user avatar
  • 33.8k
3 votes

What is the Bellman equation for V(s) in the case of a deterministic environment?

Your 2nd equation is the Bellman optimality equation (BOE) for $V$. So, to emphasise that, you could write it as follows $$ V^\color{red}{*}(s) = \max_a(R(s,a) + \gamma\sum_{s'} P(s,a,s') V^\color{red}...
user avatar
  • 33.8k
2 votes
Accepted

How can a neural network work with continuous time?

By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the ...
user avatar
  • 1,953
2 votes
Accepted

How to assign rewards in a non-Markovian environment?

Dealing with a Non-Markovian process is unusual in Reinforcement Learning. Although some explicit attempts have been made, the most common approach when confronted with a non-Markovian environment is ...
user avatar
2 votes
Accepted

Is there a way to train an RL agent without any environment?

There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the ...
user avatar
2 votes
Accepted

What are episodic and non-episodic domains in reinforcement learning?

Assuming that continuing means non terminating, what does non-episodic or episodic domain mean ? Non-episodic means the same as continuing. The quote you found is not listing two separate domains, ...
user avatar
  • 23.8k
2 votes
Accepted

How does the learning rate $\alpha$ vary in stationary and non-stationary environments?

So why is constant-$\alpha$ being used? This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the ...
user avatar
  • 23.8k
2 votes

What is the Bellman equation for V(s) in the case of a deterministic environment?

You're correct, that's the definition of the Bellman equation in the deterministic case. You can refer to the Wikipedia article of the Bellman equation where $F(x, a)$ is the reward function, with $x$ ...
user avatar
1 vote

How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?

I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which ...
user avatar
  • 705

Only top scored, non community-wiki answers of a minimum length are eligible