# Why are we using all hyperparameters in RL?

I am new in RL and I am trying to understand why do we need all these hyperparameters. Can somebody explain me why we use them and what are the best values to use for them?

total_episodes = 50000 # Total episodes

total_test_episodes = 100 # Total test episodes

max_steps = 99 # Max steps per episode

learning_rate = 0.7 # Learning rate

gamma = 0.618 # Discounting rate

# Exploration parameters

epsilon = 1.0 # Exploration rate

max_epsilon = 1.0 # Exploration probability at start

min_epsilon = 0.01 # Minimum exploration probability

decay_rate = 0.01 # Exponential decay rate

I am currently working on taxi_v2 problem from GYM.

In RL, there are episodic and non-episodic tasks (or problems). In episodic tasks, each episode proceeds in time steps.

For example, most games are episodic tasks. For instance, in a football championship (e.g. Premier League), each football match during the whole season is an episode. In this example, each minute (or second) of a football match can be considered a time step of the the episode (that is, the football match).

The parameter total_episodes thus specifies the number of episodes of your RL episodic problem. Similarly, max_steps specifies the maximum number of time steps per episode.

Why do we also need total_test_episodes? In machine learning, when building a model, there are usually two phases: the training phase and the test phases. In the training phase, you use the training dataset to learn the parameters of the model. In the test phase, you test the performance (e.g. the total return, in the case of RL) of the model. Hence, total_test_episodes is used to specify this hyper-parameter.

In deep RL, a (deep) neural network (NN) is used to represent either a value function or a policy (which is also a function). In machine learning, neural networks are usually trained using an optimisation method, like gradient descent (GD) and the back-propagation (BP), which is used to compute the gradient of the objective function with respect to the parameters of the model. In the case of deep RL, the parameters of the NN that represents either the value function or policy are also learned using a similar approach. In this context, the learning rate is a hyper-parameter that determines the "strength" of the update step of the optimisation algorithm. More concretely, in the case of GD, the update step is

$$\mathbf{\theta}_{n+1} \gets \mathbf{\theta}_{n} - \alpha \nabla f(\mathbf{\theta} _{n})$$

where $$\mathbf{\theta}_{n+1}$$ is a vector containing the parameter of your model (in this case, a NN), $$\alpha$$ is the learning rate and $$\nabla f(\mathbf{\theta} _{n})$$ is the gradient of the objective function $$f$$. Hence, learning_rate specifies the value of $$\alpha$$.

In your case, learning_rate can actually specify the value of the learning rate of your RL algorithm. For example, in the case $$Q$$-learning, the learning rate, which we can denote by $$\alpha$$, also specifies the "strength" of the update.

Similarly, the gamma is a hyper-parameter of your RL algorithm. For example, in the case $$Q$$-learning, the parameter $$\gamma$$ (the discount factor) which determines the contribution of the estimate of the $$Q$$ value of the next state (while taking the greedy action from that state) to the $$Q$$ value that is being currently updated.

$$Q$$-learning is an off-policy RL algorithm, which means that it uses a behaviour policy that is possibly different than the policy it tries to estimate. The usual behaviour policy is the $$\epsilon$$-greedy (with probability $$\epsilon$$ a random action is taken in a certain state and with probability $$1 - \epsilon$$ the greedy action is taken). The parameter epsilon specifies this hyper-parameter. In this context, an initial $$\epsilon$$ that specifies the exploration rate is provided. As the training progresses, the estimate of the optimal value function should become more accurate. In that case, you want to explore less and follow more your estimate of the optimal value function, so min_epsilon is used to specify the lowest $$\epsilon$$.

The parameter decay_rate is used to specify the value of the decay rate. Have a look at https://stats.stackexchange.com/a/31334/82135 for more info.

• @nbro Isn't gamma related to the importance you want to give to the value of the rewards in the next states? Instead of, as you said, the future values of the Q value? Or maybe that was what you wanted to say and I misunderstood it. – Miguel Saraiva Jun 13 '19 at 16:51
• @MiguelSaraiva Yes, I think I stated it wrongly. It is the value in the next state. – nbro Jun 13 '19 at 19:51