What does "adding one additional step to the policy" mean?
This is an informal shorthand for how many backup steps from a non-zero reward or learned value will receive a meaningful update. That is, an update with actual data other than the initial bootstrap.
This is due to three things:
Bootstrap updates in basic TD only look one timestep ahead, e.g. from $s,a$ to $s',a'$, with the update $Q(s, a) = Q(s, a) + \alpha(r + \gamma Q(s', a') - Q(s, a))$
Only the last step in the example environment has a non-zero reward.
Only values that have already received at least one "meaningful" update are themselves useful in bootstrapping real data back to earlier time steps. Without such an update, they only have initial arbitrary values.
With background planning, as in Dyna, there is a chance of choosing to base the update on a state, action pair that has already been updated with some non-zero* data, whilst the agent is exploring elsewhere.
Without background planning, and only using single-step backups, initially only the ends of episodes will receive updates with meaningful data.
You can also use multi-step backups, or eligibility traces - as in TD($\lambda$) - to increase the number of steps that get updated at once. These options each have their own pros and cons.
* Technically it is not non-zero that matters, but "different from other experience". Initially the agent experiences a bunch of states and actions that all look the same to it in terms of its estimates of expected future reward. Before an agent can learn to choose between actions it needs to experience results that are different between them.
