Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
1 Answer
Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
Technically yes.
The value update step in Policy Iteration is:
$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$
The discount factor $\gamma$ can be set to $0$, making the update:
$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))r$$
However, there are two key details that are important, and make this a technically yes, not some alternative way of solving problems:
Changing discount factor $\gamma$ changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.
Technically there is still a value function being updated. The function $v(s)$ is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.
Due to the lack of bootstrapping between states, all the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using $\pi(s) = \text{argmax}_a[\sum_{r,s'}p(r,s'|s,a)r]$ for any state, making a policy iteration process redundant.
However, with these caveats in mind, yes this is still policy iteration. It is the same update process, just with a particular choice of one of the parameters.
-
$\begingroup$ Here is a related question, you might find interesting! Thank you for the detailed answer here! $\endgroup$ Apr 22, 2020 at 7:25