What are the best hyper-parameters to tune in reinforcement learning?

Question

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, entropy coefficient, etc.

For example, in "normal" ML, the batch size and learning rate are typically the main hyper-parameters that get optimised first.

Specifically, I am using PPO, but this can probably be applied to a lot of other RL algorithms too.

Answer 1

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL.

It suggests that discount factor and learning rate are the two most important parameters to tune, followed by the width of the policy/value functions.

That study also reports that it's very important to normalize the observations, and initialize the policy so that the initial actions are zero mean with a very small variance.

Answer 2

Personally, I would choose the following two as the most important:

• epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too much exploration might just waste your time.
• learning rate: The learning rate determines how fast do you learn from new states of experience. A learning rate that is too high might not be good in cases when the environment has many states with high probabilities of negative rewards, i.e. many penalizations. This might make your agent move back and forth in the same place in order to avoid getting penalized. Also, a learning rate that is too low might make your agent learn very slowly and depending on your epsilon, the agent might enter a phase of exploitation with very little knowledge of an optimal policy.