1
$\begingroup$

This is my model average rewards as follow image.

How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?

And I'm using PPO to train.

enter image description here

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning from pixels such as the Atari games, you can easily expect 1-10M steps.

You can get a feeling by looking at many solved environments, for instance here: https://github.com/Rafael1s/Deep-Reinforcement-Learning-Algorithms

Improve this answer
$\endgroup$
0
$\begingroup$

It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the CleanRL benchmarks, there are a few of them where they use PPO: https://wandb.ai/openrlbenchmark/openrlbenchmark/reportlist.

Improve this answer
New contributor
kir0ul is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jun 23 at 22:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .