Are policy and value iteration used only in grid world like scenarios?

Question

I am trying to self learn reinforcement learning. At the moment I am focusing on policy and value iteration, and I am finding several problems and doubts.

One of the main doubts is given by the fact that I can't find many diversified examples on how to implement these on python, instead I find always only the classical grid world example.

So, my doubt is: Are policy and value iteration used only in grid world like scenarios, or can be used also in other contexts?

Answer 1

Policy and value iteration both require you to, for each possible transition and each corresponding possible reward at each state, compute a statistic of $r + \gamma V(s')$. In order for this to be tractable, you need for there to be at most finitely many states, actions, possible rewards, and possible transitions at each state. You also need to know the transition model. This is the case in gridworld.

Gridworld is not the only example of an MDP that can be solved with policy or value iteration, but all other examples must have finite (and small enough) state and action spaces. For example, take any MDP with a known model and bounded state and action spaces of fairly low dimension. Then you can approximate the state and action spaces with a finite number of bins, each corresponding to its own " discretized state/action". With smooth enough dynamics and enough bins, you'll be able to solve the MDP with policy/value iteration on the discretized spaces.

In many interesting RL problems though,

1. You don't know the transition model, and/or
2. The state space, action space, and/or reward space are too large

In these cases you wouldn't be able to compute the value function exactly, so you can't really do policy/value iteration. However, in most value based RL algorithms, the policy evaluation / policy improvement steps are approximated using sample transitions and function approximators.