I would like the community to help me understand if the following example would be better represented as episodic or continuous task, this will help me structure the problem and chose the right RL algorithm.

The agent start with an initial score x of let's say 100. The agent objective is to maximise it's score. There is no upper bound! Theoretically the agent can get a score up to infinity, and there is no termination based on the number of steps, therefore the agent could play forever. However, the score can't be negative and if the agent get to a score of zero, the episode should terminate and the environment reset. I am undecided what would be the best representation, because if the agent learns how to play, the episode would never terminate, and the agent would theoretically play forever. However if the score get to zero, there is no way for the agent to continue playing so the environment needs to reset. Thank you.

  • 1
    $\begingroup$ your problem is episodic as there is a terminal state. this is similar to cart pole in that a successful agent would never have an end of episode because it can perfectly balance the pole without it dropping. It may be worthwhile having an artificial stopping criteria for training — e.g. stopping after 5000 steps and restarting (but don’t count this as a termination). The number of steps would depend on your environment and how much it changes as the length of the episode goes on $\endgroup$ Dec 28 '21 at 11:54
  • $\begingroup$ Thank you for your answer! The environment is quite chaotic and ever changing, and the agent should always try to adapt to it, therefore it is not quite like the cart pole in the sense that in cart pole, once the agent has learned how to balance the pole will stay always upright. But i see your point, and i think you may be right $\endgroup$
    – DarioB
    Dec 28 '21 at 15:19
  • 1
    $\begingroup$ In which case, I would definitely recommend using some kind of TD algorithm (Actor Critic/Q-Learning) and letting episodes run as usual. Make sure to add checkpoints to save the model so you can kill it when it is sufficiently trained $\endgroup$ Dec 28 '21 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.