2
$\begingroup$

I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and the task is to apply the Q-learning algorithm to fill out a Q-table (which I've done) but confused on what it may mean by the learned policy.

So, what is a "learned policy" in Q-learning?

$\endgroup$
3
$\begingroup$

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed something about the policy.

The most common policy scenarios with Q learning are that it will converge on (learn) the values associated with a given target policy, or that it has been used iteratively to learn the values of the greedy policy with respect to its own previous values. The latter choice - using Q learning to find an optimal policy, using generalised policy iteration - is by far the most common use of it.

A policy is not a list of values, it is a map from state to actions. The question wants you to show the policy that you have learned the Q values for.

The policy in your case is therefore likely to be to pick the action that has the highest action value in each state. You may be able to decribe your answer in text ("always turn left unless next to the exit") or as a graphic (draw arrows on a grid world to show the preferred direction). Or you could write out a table of states showing the chosen action in each one.

The maths notation for how you derive the policy from a Q table can be written:

$$\pi(s) = \text{argmax}_a Q(s,a)$$

Or a bit more formally:

$$\pi: \mathcal{S} \rightarrow \mathcal{A} = \text{argmax}_{a \in \mathcal{A}(s)} Q(s,a)\qquad \forall s \in \mathcal{S}$$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy