0
$\begingroup$

I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return:

  • finite horizon undiscounted return (used for episodic problems)

$G = \sum_{t=0}^T R_t$

  • infinite horizon discounted return (used for non-episodic problems).

$G = \sum_{t=0}^\infty \gamma^t R_t$

However, I'm confusing that "Is Cartpole episodic task?". Ideally, the simulation lasts forever. This is my final objective (prolonging the lifetime). But it still has some termination states. Should I introduce the termination state and use it with a discounted return like:

$G = \sum_{t=0}^T \gamma^t R_t$

$\endgroup$
0
$\begingroup$

It's not true that finite horizon MDP can't use a discount factor. Any discount factor $\gamma \leq 1$ is fine for finite horizon.

Whether or not cartpole is a finite or infinite horizon depends on the environment implementation. The default environment in openai gym uses $T=200$ as the horizon. This is customizable, and most papers I see which use cartpole as a test environment use $T=1000$ instead. The task is considered "solved" when the agent can reach $T$ timesteps consistently (e.g., the last 100 episodes have all reached $T$ timesteps).

If you want to treat cartpole as an infinite horizon task, that's fine too. Clearly you need $\gamma < 1$ here, and you can just $T$ to some arbitrary large number.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.