3
$\begingroup$

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, entropy coefficient, etc.

For example, in "normal" ML, the batch size and learning rate are typically the main hyper-parameters that get optimised first.

Specifically, I am using PPO, but this can probably be applied to a lot of other RL algorithms too.

$\endgroup$
1
  • $\begingroup$ Maybe rather than asking "What are the best..."? Maybe you could reword your question to something like "Which hyper-parameters typically affect the performance of the RL algorithms the most, so that should be tuned first? Are there any guidelines or research on this topic?" $\endgroup$ – nbro May 29 at 0:31
3
$\begingroup$

Personally, I would choose the following two as the most important:

  • epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too much exploration might just waste your time.
  • learning rate: The learning rate determines how fast do you learn from new states of experience. A learning rate that is too high might not be good in cases when the environment has many states with high probabilities of negative rewards, i.e. many penalizations. This might make your agent move back and forth in the same place in order to avoid getting penalized. Also, a learning rate that is too low might make your agent learn very slowly and depending on your epsilon, the agent might enter a phase of exploitation with very little knowledge of an optimal policy.
$\endgroup$
1
  • 2
    $\begingroup$ @NeilSlater I agree, I had mixed feelings about rewards since it might introduce bias. Also, what could have originally been intended before "tuning" rewards, might have also been achieved by tuning the learning rate and epsilon. I will edit the answer. Thank you. $\endgroup$ – devidduma May 28 at 14:41
2
$\begingroup$

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL.

It suggests that discount factor and learning rate are the two most important parameters to tune, followed by the width of the policy/value functions.

That study also reports that it's very important to normalize the observations, and initialize the policy so that the initial actions are zero mean with a very small variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.