1
$\begingroup$

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-table for given state i.e., the action with maximum Q-value. So, in case of second player, a Q-table is required.

It is known that Q-table has to be constructed only by running several episodes after its arbitrary initialization. So, the two players has to play with some policy to construct a Q-table for player two. Since the first player uses random policy. I have doubt regarding the policy of second player for constructing Q-table.

I have doubt regarding the policy of second player only. I know the policy he need to follow after the completion of updation of Q-table. I am not sure about the policy he need to follow while updating the Q-table only.

Which policy does my second player need to follow while constructing the Q-table for himself? Can he use random policy like player one? Or does he need to use arbitrarily initialized or partially updated Q-table itself for selecting best action? Or can he use some other policy till the completion of updating Q-table?

$\endgroup$
1
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – nbro
    Jul 6 at 10:45
3
$\begingroup$

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).

Q learning is an off policy algorithm, meaning that the Q values can be learned regardless of the policy used to collect data. So the Q player can be following a random policy, or even a fixed pre defined policy if you want. Usually, however, before the Q table has converged, we want the agent to explore the environment to collect the data it needs (without sacrificing too much return, this is the exploration-exploitation tradeoff).

It is very common practice to use the intermediate Q tables to induce a policy for the agent. For example, the epsilon-greedy policy will select the best action according to the current Q table with high probability, and will otherwise choose a random action.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.