The key is that reinforcement learning through something like, say, SARSA, works by splitting up the state space into discrete points, and then trying to learn the best action at every point.
To do this, it tries to pick actions that maximize the reward signal, possibly subject to some kind of exploration policy like epsilon-greedy.
In cart-pole, two common reward signals are:
- Receive 1 reward when the pole is within a small distance of the topmost position, 0 otherwise.
- Receive a reward that linearly increases with the distance the pole is off the ground.
In both cases, an agent can continue to learn after the pole has fallen: it will just want to move the poll back up, and will try to take actions to do so.
However, an offline algorithm wouldn't update its policy while the agent is running. This kind of algorithm wouldn't benefit from a continuous task. An online algorithm, on contrast, updates its policy as it goes, and has no reason to stop between episodes, except that it might become stuck in a bad state.