In RL, there are episodic and non-episodic tasks (or problems). In episodic tasks, each episode proceeds in time steps.
For example, most games are episodic tasks. For instance, in a football championship (e.g. Premier League), each football match during the whole season is an episode. In this example, each minute (or second) of a football match can be considered a time step of the the episode (that is, the football match).
The parameter total_episodes
thus specifies the number of episodes of your RL episodic problem. Similarly, max_steps
specifies the maximum number of time steps per episode.
Why do we also need total_test_episodes
? In machine learning, when building a model, there are usually two phases: the training phase and the test phases. In the training phase, you use the training dataset to learn the parameters of the model. In the test phase, you test the performance (e.g. the total return, in the case of RL) of the model. Hence, total_test_episodes
is used to specify this hyper-parameter.
In deep RL, a (deep) neural network (NN) is used to represent either a value function or a policy (which is also a function). In machine learning, neural networks are usually trained using an optimisation method, like gradient descent (GD) and the back-propagation (BP), which is used to compute the gradient of the objective function with respect to the parameters of the model. In the case of deep RL, the parameters of the NN that represents either the value function or policy are also learned using a similar approach. In this context, the learning rate is a hyper-parameter that determines the "strength" of the update step of the optimisation algorithm. More concretely, in the case of GD, the update step is
$$\mathbf{\theta}_{n+1} \gets \mathbf{\theta}_{n} - \alpha \nabla f(\mathbf{\theta} _{n})$$
where $\mathbf{\theta}_{n+1}$ is a vector containing the parameter of your model (in this case, a NN), $\alpha$ is the learning rate and $\nabla f(\mathbf{\theta} _{n})$ is the gradient of the objective function $f$. Hence, learning_rate
specifies the value of $\alpha$.
In your case, learning_rate
can actually specify the value of the learning rate of your RL algorithm. For example, in the case $Q$-learning, the learning rate, which we can denote by $\alpha$, also specifies the "strength" of the update.
Similarly, the gamma
is a hyper-parameter of your RL algorithm. For example, in the case $Q$-learning, the parameter $\gamma$ (the discount factor) which determines the contribution of the estimate of the $Q$ value of the next state (while taking the greedy action from that state) to the $Q$ value that is being currently updated.
$Q$-learning is an off-policy RL algorithm, which means that it uses a behaviour policy that is possibly different than the policy it tries to estimate. The usual behaviour policy is the $\epsilon$-greedy (with probability $\epsilon$ a random action is taken in a certain state and with probability $1 - \epsilon$ the greedy action is taken). The parameter epsilon
specifies this hyper-parameter. In this context, an initial $\epsilon$ that specifies the exploration rate is provided. As the training progresses, the estimate of the optimal value function should become more accurate. In that case, you want to explore less and follow more your estimate of the optimal value function, so min_epsilon
is used to specify the lowest $\epsilon$.
The parameter decay_rate
is used to specify the value of the decay rate. Have a look at https://stats.stackexchange.com/a/31334/82135 for more info.