Which policy has to be followed by a player while construction of its own Q-table?

Question

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-table for given state i.e., the action with maximum Q-value. So, in case of second player, a Q-table is required.

It is known that Q-table has to be constructed only by running several episodes after its arbitrary initialization. So, the two players has to play with some policy to construct a Q-table for player two. Since the first player uses random policy. I have doubt regarding the policy of second player for constructing Q-table.

I have doubt regarding the policy of second player only. I know the policy he need to follow after the completion of updation of Q-table. I am not sure about the policy he need to follow while updating the Q-table only.

Which policy does my second player need to follow while constructing the Q-table for himself? Can he use random policy like player one? Or does he need to use arbitrarily initialized or partially updated Q-table itself for selecting best action? Or can he use some other policy till the completion of updating Q-table?

Answer 1

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).

Q learning is an off policy algorithm, meaning that the Q values can be learned regardless of the policy used to collect data. So the Q player can be following a random policy, or even a fixed pre defined policy if you want. Usually, however, before the Q table has converged, we want the agent to explore the environment to collect the data it needs (without sacrificing too much return, this is the exploration-exploitation tradeoff).

It is very common practice to use the intermediate Q tables to induce a policy for the agent. For example, the epsilon-greedy policy will select the best action according to the current Q table with high probability, and will otherwise choose a random action.