Skip to main content
added 304 characters in body
Source Link
Neil Slater
  • 33.3k
  • 3
  • 44
  • 65

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed something about the policy.

The most common policy scenarios with Q learning are that it will converge on (learn) the values associated with a given target policy, or that it has been used iteratively to learn the values of the greeygreedy policy with respect to its own previous values. The latter choice - using Q learning to find an optimal policy, using generalised policy iteration - is by far the most common use of it.

A policy is not a list of values, it is a map from state to actions. The question wants you to show the policy that you have learned the Q values for.

The policy in your case is therefore likely to be to pick the action that has the highest action value in each state. You may be able to decribe your answer in text ("always turn left unless next to the exit") or as a graphic (draw arrows on a grid world to show the preferred direction). Or you could write out a table of states showing the chosen action in each one.

The maths notation for how you derive the policy from a Q table can be written:

$$\pi(s) = \text{argmax}_a Q(s,a)$$

Or a bit more formally:

$$\pi: \mathcal{S} \rightarrow \mathcal{A} = \text{argmax}_{a \in \mathcal{A}(s)} Q(s,a)\qquad \forall s \in \mathcal{S}$$

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed something about the policy.

The most common policy scenarios with Q learning are that it will converge on (learn) the values associated with a given target policy, or that it has been used iteratively to learn the values of the greey policy with respect to its own values. The latter choice - using Q learning to find an optimal policy - is by far the most common use of it.

A policy is not a list of values, it is a map from state to actions. The question wants you to show the policy that you have learned the Q values for.

The policy in your case is therefore likely to be to pick the action that has the highest action value in each state. You may be able to decribe your answer in text ("always turn left unless next to the exit") or as a graphic (draw arrows on a grid world to show the preferred direction).

The maths notation for how you derive the policy from a Q table can be written:

$$\pi(s) = \text{argmax}_a Q(s,a)$$

Or a bit more formally:

$$\pi: \mathcal{S} \rightarrow \mathcal{A} = \text{argmax}_{a \in \mathcal{A}(s)} Q(s,a)\qquad \forall s \in \mathcal{S}$$

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed something about the policy.

The most common policy scenarios with Q learning are that it will converge on (learn) the values associated with a given target policy, or that it has been used iteratively to learn the values of the greedy policy with respect to its own previous values. The latter choice - using Q learning to find an optimal policy, using generalised policy iteration - is by far the most common use of it.

A policy is not a list of values, it is a map from state to actions. The question wants you to show the policy that you have learned the Q values for.

The policy in your case is therefore likely to be to pick the action that has the highest action value in each state. You may be able to decribe your answer in text ("always turn left unless next to the exit") or as a graphic (draw arrows on a grid world to show the preferred direction). Or you could write out a table of states showing the chosen action in each one.

The maths notation for how you derive the policy from a Q table can be written:

$$\pi(s) = \text{argmax}_a Q(s,a)$$

Or a bit more formally:

$$\pi: \mathcal{S} \rightarrow \mathcal{A} = \text{argmax}_{a \in \mathcal{A}(s)} Q(s,a)\qquad \forall s \in \mathcal{S}$$

Source Link
Neil Slater
  • 33.3k
  • 3
  • 44
  • 65

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed something about the policy.

The most common policy scenarios with Q learning are that it will converge on (learn) the values associated with a given target policy, or that it has been used iteratively to learn the values of the greey policy with respect to its own values. The latter choice - using Q learning to find an optimal policy - is by far the most common use of it.

A policy is not a list of values, it is a map from state to actions. The question wants you to show the policy that you have learned the Q values for.

The policy in your case is therefore likely to be to pick the action that has the highest action value in each state. You may be able to decribe your answer in text ("always turn left unless next to the exit") or as a graphic (draw arrows on a grid world to show the preferred direction).

The maths notation for how you derive the policy from a Q table can be written:

$$\pi(s) = \text{argmax}_a Q(s,a)$$

Or a bit more formally:

$$\pi: \mathcal{S} \rightarrow \mathcal{A} = \text{argmax}_{a \in \mathcal{A}(s)} Q(s,a)\qquad \forall s \in \mathcal{S}$$