I would like to use OpenAI Gym to solve a continuing environment, that is, a problem with a single, never-ending episode (please note I don't mean a continuous environment with continuous state and actions).

The only continuing environment I found in their repository was the classic inverted pendulum problem, and I found no baseline methods (algorithms) that don't require episodic environments.

So I have two questions:

  • are there any continuing environments other than the inverted pendulum one?

  • is there an OpenAI Gym baseline method that I can use to solve the inverted pendulum problem as well as other continuing environments?


I'm not sure why you need a continuing environment, but actually you can make most (if not all) OpenAI Gym environments continuing. When you perform a step, you receive information about the next state, the reward, a termination signal and a dictionary with additional information. Simply ignore the termination signal if you want the episodes to continue indefinitely. In some cases you will need to modify a variable of the environments called $\texttt {_max_episode_steps}$, which may force the simulation to stop or to restart.

About your second question, check a resource called Spinning Up, from OpenAI too. They explain several methods and their implementation.

  • $\begingroup$ For most OpenAI gym environments, your suggested modifications will unconstrain the episode lengths, but that is not the same as making them continuous. E.g. this won't make MountainCar or LunarLander continuous - they are still goal driven to a terminal state, and any successful agent will solve them in that way. $\endgroup$ – Neil Slater Dec 9 '19 at 7:49
  • $\begingroup$ I don't understand your point. The inverted pendulum is also driven to a terminal state, but can be understood as a continuing problem. What is the difference? Nonetheless, if you do these modifications to MuJoCo environments, they certainly will be continuing environments with no terminal state. $\endgroup$ – Diego Gomez Dec 9 '19 at 21:47
  • $\begingroup$ Thanks! Spinning Up is cool, why on earth don't they point to it on the OpenAI Gym main pages? $\endgroup$ – user118967 Dec 10 '19 at 4:17
  • $\begingroup$ I need a continuing environment because I am working on a problem with one, although now I realize that I can still train them using a time limit and the use them in the continuing scenario. $\endgroup$ – user118967 Dec 10 '19 at 4:18
  • 1
    $\begingroup$ Regarding Neil Slater's point, I think he means using your trick on environments that have a natural termination may be tricky. For example, once the mountain car is on top or the lunar lander is crashed, what does even mean to continue the episode? I was asking about environments that are naturally continuing and that can keep going indefinitely without large quantitative changes like reaching the mountain top or landing. $\endgroup$ – user118967 Dec 10 '19 at 4:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.