# Tag Info

10

I think the distinction is made more for conceptual reasons, which has practical implications, so let me review the usual definitions of a stochastic and partially observable environment. A stochastic environment can be modeled as a Markov Decision Process (MDP) or Partially Observable MDP (POMDP). So, an environment can be stochastic and partially ...

8

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an effect on the racket. Here's a better way: env.unwrapped.get_action_meanings()

8

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally deterministic unless: Important state information is missing (a POMDP). For example, in a map where the agent is not allowed to know its exact location or remember ...

6

A few points I'd like to add (without repeating the info already provided by nbro's answer): I think you're half-right, in that indeed we can probably always model randomness as hidden information (e.g., as the hidden random seed in a software implementation of an environment). However, the other way around does not work; we can not always model any ...

5

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms. MDP vs POMDP In Markovian Decision Processes (MDPs) the whole state space is known, this means you know all the information for your problem; therefore, you can use them to find ...

5

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and $n$. This is a simple stochastic environment: this is a one state environment, but it is still an environment. But obviously the optimal policy is to choose ...

4

You can try to figure out what exactly does an action do using such script: action = 0 # modify this! o = env.reset() for i in xrange(5): # repeat one action for five times o = env.step(action)[0] IPython.display.display( Image.fromarray( o[:,140:142] # extract your bat ).resize((300, 300)) # bigger image, easy for visualization ) ...

4

This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment. Yes you will very likely need to write software to represent your environment in some standard way as a Reinforcement Learning (RL) environment. Depending on details, this may be trivially easy or it might be quite involved. An environment in RL ...

4

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem There is no difference in concept, which is why tic-tac-toe and maze problems are used to teach. As you have noted, the main difference between reinforcement learning (RL) and supervised learning is that RL does not use labeled datasets. If you ...

4

You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like this is not a fully observable environment. I did not use the term "fully observable game" or "fully observable system" above , because ...

3

You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem and $\mathcal{I}$ to be the set of possible items. Provided both of these sets are finite, then you should still be able to approach this problem with DQN. ...

3

If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an end of episode, which assumes a return of 0 from any terminal state $S_{T}$. Of course that is because it is not the end of the episode. You have two "...

3

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely uninformative. This is why a stochastic policy is what is needed. This is common for POMDPs, whereas for regular MDPs we can always find a deterministic policy ...

3

To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki: The episode ends when you reach 0.5 position, or if 200 iterations are reached. There is a deeper question in what you asked, though: My initial understanding was that an episode should end when the Car reaches the flagpost. The environment certainly ...

3

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the _max_episode_steps attribute of the environment.

3

In reinforcement learning, is the value of terminal/goal state always zero? Yes, always for episodic problems, the value of a terminal state is always zero, from the definition. The value of a state $v(s)$ is the expected sum (perhaps discounted) of rewards from all future time steps. There are no future time steps when in a terminal state, so this sum must ...

3

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case? The learned $Q$-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according ...

3

There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment. "Each action is repeatedly performed for a duration of k frames, where k is uniformly sampled from {2,3,4}" So the action is just repeated a different number of times due to randomness

3

First, note that the current state does not determine the next state. What determines the next state are the dynamics of the environment, which, in the context of reinforcement learning and, in particular, MDPs, are encoded in the probability distribution $p(s', r \mid s, a)$. So, if the agent is in a certain state $s$, it could end up in another state $s'$, ...

2

By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the outside environment is acting, rather than how your code is taking its steps. These are some lines from the book, Artificial Intelligence: A Modern Approach: ...

2

There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational DQN paper is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple ...

2

Assuming that continuing means non terminating, what does non-episodic or episodic domain mean ? Non-episodic means the same as continuing. The quote you found is not listing two separate domains, the word "continuing" is slightly redundant. I expect the author put it in there to emphasise the meaning, or to cover two common ways of describing such ...

2

Dealing with a Non-Markovian process is unusual in Reinforcement Learning. Although some explicit attempts have been made, the most common approach when confronted with a non-Markovian environment is to try and make the agent's representation of it Markovian. After reducing Agent's model of the dynamics to a Markovian process, rewards are assigned from the ...

2

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward. I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a ...

2

So why is constant-$\alpha$ being used? This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly. From my understanding, in stationary environments, ...

1

I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent. I believe that this varying number of applicable observations (depending on the number of currently present agents) ...

1

PEAS stands for (Performance, Environment, Actuators and Sensors), when you are asked to give the peas of a AI then you should describe it as follows: Example: PEAS for refinery controller: • Performance measure: maximize purity, yield, safety • Environment: refinery, operators • Actuators: valves, pumps, heaters, displays • Sensors: temperature, ...

1

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.

1

I guess it would always be better if you can reuse existing environments to make it work for yourself. Since most of the environment codes is anyway opensourced, you can always edit it to your liking. If you want a custom environment, you can add an environment to gym like this.

Only top voted, non community-wiki answers of a minimum length are eligible