2
$\begingroup$

Value iteration

This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal policy.

Is using Q learning for control the same for finding optimal policy? Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?

$\endgroup$

1 Answer 1

3
$\begingroup$

Q learning is very similar to value iteration. They are based on the same principles. A key similarity is that both assume a greedy action choice on the bootstrap next state value.

The big difference is that value iteration is based on processing all states and possible actions whilst knowing the MDP model in detail. For Q learning, the value function is instead learned by sampling from experience, and it does not require knowledge of the MDP model, beyond the state and action representations.

Q learning typically uses an $\epsilon$-greedy policy derived from its current action value function estimate, in order to drive exploration and take samples. Whilst value iteration neither explores nor exploits nor samples, instead resolving based on the model, processing all states and actions. Value iteration can be thought of as a kind of planning algorithm.

Or do I have to perform generalized policy iteration where q learning is for evaluation and some other method for improvement till convergence?

Q learning is already a form of generalised policy iteration (GPI). GPI is the underlying principle behind value-based control methods. You can think of policy iteration (and value iteration) as the "standard" for accuracy, but it doesn't scale well or apply directly when the model is not known. GPI is an abstract view of how learning values and then basing policies on greedily selecting on those values should eventually converge on an optimal policy.

The difference between using off policy TD learning for evaluation or control is in how you select policies. For evaluation you would have a fixed target policy used to calculate bootstrap values (for TD error or TD target). For control scenarios, the target policy is always the greedy action choice based on current value estimates, which is a policy that changes over time. Q learning is exclusively a TD control method. It is easy to adapt it to create an evaluation method, but AFAIK that would not be called Q learning.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .