Can policy iteration use only the immediate reward for updates?

Question

Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?

Answer 1

Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?

Technically yes.

The value update step in Policy Iteration is:

$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$

The discount factor $\gamma$ can be set to $0$, making the update:

$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))r$$

However, there are two key details that are important, and make this a technically yes, not some alternative way of solving problems:

• Changing discount factor $\gamma$ changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.

• Technically there is still a value function being updated. The function $v(s)$ is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.

• Due to the lack of bootstrapping between states, all the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using $\pi(s) = \text{argmax}_a[\sum_{r,s'}p(r,s'|s,a)r]$ for any state, making a policy iteration process redundant.

However, with these caveats in mind, yes this is still policy iteration. It is the same update process, just with a particular choice of one of the parameters.