Policy and value iteration both require you to, for each possible transition and each corresponding possible reward at each state, compute a statistic of $r + \gamma V(s')$. In order for this to be tractable, you need for there to be at most finitely many states, actions, possible rewards, and possible transitions at each state. You also need to know the transition model. This is the case in gridworld.
Gridworld is not the only example of an MDP that can be solved with policy or value iteration, but all other examples must have finite (and small enough) state and action spaces. For example, take any MDP with a known model and bounded state and action spaces of fairly low dimension. Then you can approximate the state and action spaces with a finite number of bins, each corresponding to its own " discretized state/action". With smooth enough dynamics and enough bins, you'll be able to solve the MDP with policy/value iteration on the discretized spaces.
In many interesting RL problems though,
- You don't know the transition model, and/or
- The state space, action space, and/or reward space are too large
In these cases you wouldn't be able to compute the value function exactly, so you can't really do policy/value iteration. However, in most value based RL algorithms, the policy evaluation / policy improvement steps are approximated using sample transitions and function approximators.