# RL: how can the cart-pole problem be a continuing task?

In Introduction to Reinforcement Learning (2ed), Sutton and Barto, there is an example of Pole-Balancing problem (Example 3.4).

In this example, it said, this problem can be treated with 'episodic task' and 'continuing task'.

I think that it can only be treated as episodic task because it has an end of playing, which is falling the rod.

I have no idea how this can be treated as continuing task.... Even in OpenAI Gym cartpole env, there is an only episodic mode.

• If you could balance the rod indefinitely then it would be continuing – Gaius Aug 4 '18 at 9:36

The key is that reinforcement learning through something like, say, SARSA, works by splitting up the state space into discrete points, and then trying to learn the best action at every point.

To do this, it tries to pick actions that maximize the reward signal, possibly subject to some kind of exploration policy like epsilon-greedy.

In cart-pole, two common reward signals are:

1. Receive 1 reward when the pole is within a small distance of the topmost position, 0 otherwise.
2. Receive a reward that linearly increases with the distance the pole is off the ground.

In both cases, an agent can continue to learn after the pole has fallen: it will just want to move the poll back up, and will try to take actions to do so.

However, an offline algorithm wouldn't update its policy while the agent is running. This kind of algorithm wouldn't benefit from a continuous task. An online algorithm, on contrast, updates its policy as it goes, and has no reason to stop between episodes, except that it might become stuck in a bad state.

• I just want to know "how can cartpole environment be continuing task even though it has an end of episode"... – user3595632 Oct 19 '18 at 7:43
• end of episode = terminal state = falling off the poll – user3595632 Oct 19 '18 at 7:51
• @user3595632 As mentioned in my answer, in the continuing version of the task, the episode does not end when the pole falls. With a suitable reward function, the agent can learn to rebalance the pole after it has fallen. – John Doucette Oct 19 '18 at 19:25
• Then what happens to pole after it fallen?....Does it keep staying fallen state? – user3595632 Oct 19 '18 at 23:57
• The state of the poll is real-valued (usually represented as an angle). When it falls, it leaves a narrow region that is defined as "balanced" (note, there isn't usually a single unique balanced state, it's a range of angle values). There's nothing special about this from the perspective of the state of the poll: it's just got a different angle than before. The angle will continue to change with the effects of gravity, the kinetic energy of the pole, and the force exerted by the agent's motor. The agent will receive a different (worse) reward until the pole re-enters the "balanced" region. – John Doucette Oct 20 '18 at 16:26

Straight from Sutton's book (http://incompleteideas.net/book/the-book-2nd.html):

Example 3.4: Pole-Balancing The objective in this task is to apply forces to a cart moving along a track so as to keep a pole hinged to the cart from falling over: A failure is said to occur if the pole falls past a given angle from vertical or if the cart runs off the track. The pole is reset to vertical after each failure. This task could be treated as episodic, where the natural episodes are the repeated attempts to balance the pole. The reward, in this case, could be +1 for every time step on which failure did not occur, so that the return at each time would be the number of steps until failure. In this case, successful balancing forever would mean a return of infinity. Alternatively, we could treat pole-balancing as a continuing task, using discounting. In this case, the reward would be -1 on each failure and zero at all other times. The return at each time would then be related to K, where K is the number of time steps before failure. In either case, the return is maximized by keeping the pole balanced for as long as possible.

It's a continuing task in that, after failure, the agent always gets a reward of $$0$$ at each time-step ad infinitum.

From the book:

we could treat pole-balancing as a continuing task, using discounting. In this case the reward would be -1 on each failure and zero at all other times. The return at each time would then be related to $$-\gamma^K$$, where $$K$$ is the number of time steps before failure.

(Here I have used $$\gamma$$ as the discount factor).

Said another way, assuming the agent fails in the (K + 1)th step the reward is $$0$$ till that step, $$-1$$ for it, and then $$0$$ for eternity.

So the return: $$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + ... + \gamma^K R_{t+K+1} + ... = -\gamma^K$$