# Tag Info

7

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an effect on the racket. Here's a better way: env.unwrapped.get_action_meanings()

7

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally deterministic unless: Important state information is missing (a POMDP). For example, in a map where the agent is not allowed to know its exact location or remember ...

5

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and $n$. This is a simple stochastic environment: this is a one state environment, but it is still an environment. But obviously the optimal policy is to choose ...

5

Although I am not aware of any "benchmark problems" for (discrete) MDPs, I'll comment a bit on possible benchmarks and I will show some benchmarks used to test POMDP algorithms. MDP vs POMDP In Markovian Decision Processes (MDPs) the whole state space is known, this means you know all the information for your problem; therefore, you can use them to find ...

4

This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment. Yes you will very likely need to write software to represent your environment in some standard way as a Reinforcement Learning (RL) environment. Depending on details, this may be trivially easy or it might be quite involved. An environment in RL ...

4

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem There is no difference in concept, which is why tic-tac-toe and maze problems are used to teach. As you have noted, the main difference between reinforcement learning (RL) and supervised learning is that RL does not use labeled datasets. If you ...

3

If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an end of episode, which assumes a return of 0 from any terminal state $S_{T}$. Of course that is because it is not the end of the episode. You have two "...

3

The game of TIC-TAC-TOE can be modelled as a non-deterministic Markov decision process (MDP) if, and only if: The opponent is considered part of the environment. This is a reasonable approach when the goal is to solve playing against a specific opponent. The opponent is using a stochastic policy. Stochastic policies are a generalisation that include ...

3

You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem and $\mathcal{I}$ to be the set of possible items. Provided both of these sets are finite, then you should still be able to approach this problem with DQN. ...

3

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely uninformative. This is why a stochastic policy is what is needed. This is common for POMDPs, whereas for regular MDPs we can always find a deterministic policy ...

3

To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki: The episode ends when you reach 0.5 position, or if 200 iterations are reached. There is a deeper question in what you asked, though: My initial understanding was that an episode should end when the Car reaches the flagpost. The environment certainly ...

3

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the _max_episode_steps attribute of the environment.

3

You can try to figure out what exactly does an action do using such script: action = 0 # modify this! o = env.reset() for i in xrange(5): # repeat one action for five times o = env.step(action)[0] IPython.display.display( Image.fromarray( o[:,140:142] # extract your bat ).resize((300, 300)) # bigger image, easy for visualization ) ...

3

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case? The learned $Q$-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according ...

2

By the way you have explained things above, it seems more like a problem with your code and not the something to do with the environment. The term discrete and continuous is used to define, how the outside environment is acting, rather than how your code is taking its steps. These are some lines from the book, Artificial Intelligence: A Modern Approach: ...

2

There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment. "Each action is repeatedly performed for a duration of k frames, where k is uniformly sampled from {2,3,4}" So the action is just repeated a different number of times due to randomness

2

There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational DQN paper is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple ...

2

Assuming that continuing means non terminating, what does non-episodic or episodic domain mean ? Non-episodic means the same as continuing. The quote you found is not listing two separate domains, the word "continuing" is slightly redundant. I expect the author put it in there to emphasise the meaning, or to cover two common ways of describing such ...

2

In reinforcement learning, is the value of terminal/goal state always zero? Yes, always for episodic problems, the value of a terminal state is always zero, from the definition. The value of a state $v(s)$ is the expected sum (perhaps discounted) of rewards from all future time steps. There are no future time steps when in a terminal state, so this sum must ...

2

Dealing with a Non-Markovian process is unusual in Reinforcement Learning. Although some explicit attempts have been made, the most common approach when confronted with a non-Markovian environment is to try and make the agent's representation of it Markovian. After reducing Agent's model of the dynamics to a Markovian process, rewards are assigned from the ...

1

In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert? It is difficult to make a hard rule here. If the resulting environments can be cleanly sorted into a few different categories, and the ideal behaviour and/or the states visited are radically different in each scenario, then maybe a ...

1

In toy problems like the Short Corridor task, you can choose the state representation to explore a key property, such as the ability of a particular method to solve it. Often this is done to extremes and heavily simplified. That is what is going on here. The state space that the agent is allowed to use is made highly degenerate with respect to the problem. ...

1

I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent. I believe that this varying number of applicable observations (depending on the number of currently present agents) ...

1

Some of the domains in the International Probabilistic Planning Competition, such as the Wildlife Preserve benchmark, fit quite well the constraints you have given. Note that the problems are modeled with a high-level declarative language, RDDL. This means that you can define problems as big or as small as your heart desires with relative ease, since you can ...

1

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.

1

I guess it would always be better if you can reuse existing environments to make it work for yourself. Since most of the environment codes is anyway opensourced, you can always edit it to your liking. If you want a custom environment, you can add an environment to gym like this.

1

I've actually implemented this game before using deep reinforcement learning. You are dealing with a dynamic action space here, where the action space may change at each time step of the game (or more generally the MDP). First, let's discuss the actual action spaces in each one of the two phases of Crib (or Cribbage) and formalize the question. Phase 1: The ...

1

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward. I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a ...

1

What is the advantage of using more than one environment with a single agent? There are two main advantages to this approach: The dataset for training is closer to the independent, identically distributed (i.i.d.) ideal, important for theoretical and practical reasons when training a neural network. Samples taken from a single trajectory are not ...

1

Markov Environment is not about deterministic or stochastic. "Depends only on the current state and your action" does not mean you know what will happen(deterministic). We can have Markov + deterministic, Markov + Stochastic, Non-Markov + deterministic, and Non-Markov + stochastic. The definition you have is not a definition of deterministic. It is a ...

Only top voted, non community-wiki answers of a minimum length are eligible