When using a trained Q-learning algorithm in an actual game, would I just use exploitation and no longer use exploration? Should I use exploration only during the training phase?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Once you have estimated the $Q$ function, you can derive the policy from it in different ways. For example, you can act greedily with respect to it (see this answer), which can be formally denoted as
$$ \pi(s) = \operatorname{argmax}_{a^*}Q(s, a), \; \forall s \in \mathcal{S} $$ where $Q(s, a)$ is your estimated value function and $\pi$ the policy greedily derived from it.
This means that you would just exploit your current knowledge of the return. This is probably a good thing to do if you believe your value function is optimal and the dynamics of the environment don't change.
Of course, if your policy is not optimal, you may not want to always execute the greedy action. In that case, you could still perform some form of exploration (e.g. with the $\epsilon$-greedy policy).
Moreover, if the dynamics (e.g. the reward function) of your environment change over time, you could continually train your RL agent. If you are interested in continual RL, the paper Continual Reinforcement Learning with Complex Synapses (2018) could potentially be useful.