Why do all states appear identical under the function approximation in the Short Corridor task?

Question

This is the Short Corridor problem taken from the Sutton & Barto book. Here it's written:

The problem is difficult because all the states appear identical under the function approximation

But this doesn't make much sense as we can always choose states as 0,1,2 and corresponding feature vectors as

x(S = 0,right) = [1 0 0 0 0 0]
x(S = 0 , left) = [0 1 0 0 0 0]
x(S = 1,right) = [0 0 1 0 0 0]
x(S = 1 , left) = [0 0 0 1 0 0]
x(S = 2,right) = [0 0 0 0 1 0]
x(S = 2 , left) = [0 0 0 0 0 1]\

So why is it written that all the states appear identical under the function approximation?

Answer 1

You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely uninformative.

This is why a stochastic policy is what is needed. This is common for POMDPs, whereas for regular MDPs we can always find a deterministic policy that is guaranteed to be optimal.

Answer 2

In toy problems like the Short Corridor task, you can choose the state representation to explore a key property, such as the ability of a particular method to solve it. Often this is done to extremes and heavily simplified.

That is what is going on here. The state space that the agent is allowed to use is made highly degenerate with respect to the problem. This stands in for perhaps more complex partially observable systems, but in a way that is really clear to the reader. Also, it is still possible to derive analytically what the best policy should be, so methods can be examined as to how well they deal with the core issue (here, that state data is ambiguous).