3
$\begingroup$

This is my model average rewards as follow image.

How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?

And I'm using PPO to train.

enter image description here

$\endgroup$

2 Answers 2

1
$\begingroup$

This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning from pixels such as the Atari games, you can easily expect 1-10M steps.

You can get a feeling by looking at many solved environments, for instance here: https://github.com/Rafael1s/Deep-Reinforcement-Learning-Algorithms

$\endgroup$
0
$\begingroup$

It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the CleanRL benchmarks, there are a few of them where they use PPO: https://wandb.ai/openrlbenchmark/openrlbenchmark/reportlist.

$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jun 23, 2022 at 22:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .