Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
8 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an ...
ComputerScientist's user avatar
8 votes

Deep Q-Learning "catastrophic drop" reasons?

This is a case of overfitting the Q function leading to compounding errors when selecting actions. You have been training your policy for too long on the same data distribution. Overfitting Q ...
devidduma's user avatar
  • 552
6 votes
Accepted

Why did the openai's gym website close?

According to Open AI's Greg Brockman, the Gym website never had a big impact and so was never maintained. This is the reason he gives for shutting down the website. A read only export of the site ...
Seth Simba's user avatar
  • 1,186
6 votes
Accepted

How do you deal with movement inertia in an environment after a step?

The simplest answer is that the inertia (or velocity, if the mass of the ball is not a variable) should be part of the observable state that the agent has access to. In RL there are usually two parts ...
Neil Slater's user avatar
  • 32.4k
5 votes
Accepted

How powerful is OpenAI's Gym and Universe in board games area?

OpenAI's Gym is a standardised API, useful for reinforcement learning, applied to a range of interesting environments many of which you can then access for free with little effort. It is very simple ...
Neil Slater's user avatar
  • 32.4k
5 votes
Accepted

Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?

You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs: https://github.com/...
parrowdice's user avatar
5 votes
Accepted

Finding the true Q-values in gymnaiusm

In simple environments, the gold standard for true values to arbitrary accuracy would be to use dynamic programming, either policy iteration or value iteration (to evaluate a fixed policy, then use a ...
Neil Slater's user avatar
  • 32.4k
4 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try to figure out what exactly does an action do using such script: ...
Icyblade's user avatar
  • 141
4 votes

How do you deal with movement inertia in an environment after a step?

In RL usually a "discretized" time MDP is used in case of continuous time problems, with the idea that if we evaluate enough time the policy, we can approximate it the continuous case. This ...
Alberto's user avatar
  • 2,133
3 votes

How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

To answer your question, the specifics of some of the OpenAI Gym environments can be found on their wiki: The episode ends when you reach 0.5 position, or if 200 iterations are reached. There is a ...
DeepQZero's user avatar
  • 1,399
3 votes
Accepted

How does an episode end in OpenAI Gym's "MountainCar-v0" environment?

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the ...
harwiltz's user avatar
  • 1,136
3 votes
Accepted

My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem

There is a really small mistake in here that causes the problem: ...
hal9000's user avatar
  • 369
3 votes
Accepted

OpenAI Gym: Multiple actions in one step

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own ...
stefanbschneider's user avatar
3 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment. "Each action is repeatedly performed for a ...
jaidmin's user avatar
  • 31
3 votes
Accepted

What kind of observation state would you give for that environment?

Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem ...
Neil Slater's user avatar
  • 32.4k
2 votes

What is the difference between A2C and running an agent in an environment vector?

I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different ...
Brale's user avatar
  • 2,406
2 votes

Should I always start from the same start state in reinforcement learning?

It is your choice. This can even be different between training and target system. The approach called "exploring starts" chooses a random start state (and action if you are assessing a deterministic ...
Neil Slater's user avatar
  • 32.4k
2 votes
Accepted

Should I always start from the same start state in reinforcement learning?

It depends on the task the agent is trying to learn and of course on the environment constrains. In an Atari game agents have a pre-fixed starting point because that's part of the games rules, so I ...
Edoardo Guerriero's user avatar
2 votes
Accepted

How to define an action space when an agent can take multiple sub-actions in a step?

One way to handle an arbitrarily large sequence is by adding a STOP signal as one possible token in the sequence, just like LSTM. So you could divide your game in turns: What you now call a single ...
Andre Goulart's user avatar
2 votes

Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

Since the environment has some randomness in it, purely memorizing a trajectory to victory will not work. You will have to memorize every single trajectory for that to work, and there are an infinite ...
Sacha Gunaratne's user avatar
2 votes

Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

What's exactly the point of time.sleep() in this code? I don't really understand it, you're simply stopping the execution of the program for $0.01$ seconds, how ...
Brale's user avatar
  • 2,406
2 votes

What are the state-of-the-art results in OpenAI's gym environments?

There is the leaderboard page at the gym GitHub repository that contains links to specific implementations that "solve" the different gym environments, where "to solve" means "...
nbro's user avatar
  • 40.8k
2 votes
Accepted

What is the mapping between actions and numbers in OpenAI's gym?

There is no way to tell via the Gym API, and to any RL-based learning agent this is entirely unimportant, discrete actions are just arbitrary labels, and their effects are learned by trial and error ...
Neil Slater's user avatar
  • 32.4k
2 votes
Accepted

What is a RAM state in the gym's breakout-ram environment?

Yes, it is the state of the memory; this would mainly involve variables, since the code would be in ROM. Since it is only 128 bytes in size, the screen memory would also not be included in this. The ...
Oliver Mason's user avatar
  • 5,387
2 votes
Accepted

DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge: Sampled the noise from a standard normal distribution instead of ...
Vedant Shah's user avatar
2 votes

How do I get started with multi-agent reinforcement learning?

After checking the Internet, you will probably find several resources such as https://github.com/mohammadasghari/dqn-multi-agent-rl https://rlss.inria.fr/files/2019/07/RLSS_Multiagent.pdf https://...
penkovsky's user avatar
  • 264
2 votes

Why is training longer not better in reinforcement learning?

Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
Albert Rial's user avatar
2 votes
Accepted

Should I make my environment with gym or gymnasium?

As you correctly pointed out, OpenAI Gym is less supported these days. It makes sense to go with Gymnasium, which is by the way developed by a non-profit organization. Regarding backwards ...
penkovsky's user avatar
  • 264
2 votes

OpenAI Gym implementation of the delayed rewards

I think what you have here (with an important caveat, which I will get to later) is a common misunderstanding about how rewards should be structured for a reinforcement learning (RL) problem. It is ...
Neil Slater's user avatar
  • 32.4k
1 vote

How can I change observation states' values in OpenAI gym's cartpole environment?

I don't recommend changing the rules of the environment. What you could do: Perform a method called bucketing i.e. take a value from a continuous state space see which discrete bucket it should go ...
rert588's user avatar
  • 320

Only top scored, non community-wiki answers of a minimum length are eligible