3
$\begingroup$

I would like to use OpenAI Gym to solve a continuing environment, that is, a problem with a single, never-ending episode (please note I don't mean a continuous environment with continuous state and actions).

The only continuing environment I found in their repository was the classic inverted pendulum problem, and I found no baseline methods (algorithms) that don't require episodic environments.

So I have two questions:

  • are there any continuing environments other than the inverted pendulum one?

  • is there an OpenAI Gym baseline method that I can use to solve the inverted pendulum problem as well as other continuing environments?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I'm not sure why you need a continuing environment, but actually you can make most (if not all) OpenAI Gym environments continuing. When you perform a step, you receive information about the next state, the reward, a termination signal and a dictionary with additional information. Simply ignore the termination signal if you want the episodes to continue indefinitely. In some cases you will need to modify a variable of the environments called $\texttt {_max_episode_steps}$, which may force the simulation to stop or to restart.

About your second question, check a resource called Spinning Up, from OpenAI too. They explain several methods and their implementation.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ For most OpenAI gym environments, your suggested modifications will unconstrain the episode lengths, but that is not the same as making them continuous. E.g. this won't make MountainCar or LunarLander continuous - they are still goal driven to a terminal state, and any successful agent will solve them in that way. $\endgroup$
    – Neil Slater
    Commented Dec 9, 2019 at 7:49
  • $\begingroup$ I don't understand your point. The inverted pendulum is also driven to a terminal state, but can be understood as a continuing problem. What is the difference? Nonetheless, if you do these modifications to MuJoCo environments, they certainly will be continuing environments with no terminal state. $\endgroup$
    – Diego Gomez
    Commented Dec 9, 2019 at 21:47
  • $\begingroup$ Thanks! Spinning Up is cool, why on earth don't they point to it on the OpenAI Gym main pages? $\endgroup$
    – user118967
    Commented Dec 10, 2019 at 4:17
  • 3
    $\begingroup$ Regarding Neil Slater's point, I think he means using your trick on environments that have a natural termination may be tricky. For example, once the mountain car is on top or the lunar lander is crashed, what does even mean to continue the episode? I was asking about environments that are naturally continuing and that can keep going indefinitely without large quantitative changes like reaching the mountain top or landing. $\endgroup$
    – user118967
    Commented Dec 10, 2019 at 4:20
  • 1
    $\begingroup$ @DiegoGomez Yes user118967 has stated the same concern, not this is not addressed in your answer. I.e. you cannot turn an episodic problem into a continuous one simply by allowing more time steps. $\endgroup$
    – Neil Slater
    Commented Dec 10, 2019 at 6:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .