# Tag Info

Accepted

### Is there a fundamental difference between an environment being stochastic and being partially observable?

I think the distinction is made more for conceptual reasons, which has practical implications, so let me review the usual definitions of a stochastic and partially observable environment. A stochastic ...
• 40.6k

### Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally ...
• 32.1k

### What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an ...

### Is there a fundamental difference between an environment being stochastic and being partially observable?

A few points I'd like to add (without repeating the info already provided by nbro's answer): I think you're half-right, in that indeed we can probably always model randomness as hidden information (e....
• 10.3k

### How does Q-learning work in stochastic environments?

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case? The ...
• 10.3k

### Is the optimal policy always stochastic if the environment is also stochastic?

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
• 271
Accepted

### How do you deal with movement inertia in an environment after a step?

The simplest answer is that the inertia (or velocity, if the mass of the ball is not a variable) should be part of the observable state that the agent has access to. In RL there are usually two parts ...
• 32.1k
Accepted

### Benchmarks for reinforcement learning in discrete MDPs

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms. MDP vs POMDP In ...
• 375
Accepted

### How to deal with changing environment in reinforcement learning

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle? How does the agent know that it is ...
• 189

### What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try to figure out what exactly does an action do using such script: ...
• 141

### How to create a custom environment for reinforcement learning

This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment. Yes you will very likely need to write software to represent your environment in some ...
• 32.1k
Accepted

### Reinforcement learning with action consisting of two discrete values

You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem ...
• 4,910
Accepted

### Why do all states appear identical under the function approximation in the Short Corridor task?

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely ...
Accepted

### How should I generate datasets for a SARSA agent when the environment is not simple?

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem There is no difference in concept, which is why tic-tac-toe and maze problems are used ...
• 32.1k

### What exactly are partially observable environments?

You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like ...
• 32.1k

### How do you deal with movement inertia in an environment after a step?

In RL usually a "discretized" time MDP is used in case of continuous time problems, with the idea that if we evaluate enough time the policy, we can approximate it the continuous case. This ...
• 1,938

### How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki: The episode ends when you reach 0.5 position, or if 200 iterations are reached. There is a ...
• 1,379
Accepted

### How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the ...
• 1,136
Accepted

### How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an ...
• 32.1k
Accepted

Accepted

### What kind of observation state would you give for that environment?

Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem ...
• 32.1k

### Are all fully observable environments episodic?

No, not all fully observable environments are episodic. Let's take a look again at the definitions from the book: Fully Observable Environment (section 2.3.2) If an agent’s sensors give it access to ...
• 271
Accepted

### How can a neural network work with continuous time?

By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the ...
• 2,023

### Can non-Markov environments also be deterministic?

Markov Environment is not about deterministic or stochastic. "Depends only on the current state and your action" does not mean you know what will happen(deterministic). We can have Markov + ...
• 151