For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to van Hasselt from his book, this discretization changes the problem into a partially observable MDP (POMDP), and this is understandable. And I know POMDPs require special treatment from the vanilla Q-learning we are used to (observation space, belief states, etc).

But my question is: is there a specific technical reason why a discretized-state problem (which is now POMDP) should be solved using POMDP algorithms, instead of plainly constructing a vanilla Q-table using the discretized states (i.e. the buckets from discretization)? In other words, is there a disadvantage in not using POMDP algorithms to tackle the discretized-state problem?

  • $\begingroup$ I don't see why would state discretization that you propose cause your MDP to be partially observable, do you mind clarifying your conclusion ? $\endgroup$
    – Brale
    Feb 25 '21 at 19:35
  • $\begingroup$ When you discretize, several true states will be observed as the same. For instance, if we discretize all numbers from 0 to 0.1 as state 0, then 0.05 will be observed as state 0, and so will 0.097. $\endgroup$
    – O'Jhene
    Feb 25 '21 at 19:38
  • $\begingroup$ That's more of a problem of insufficient discretization than the method itself being inherently flawed. You can always refine your discretization until your policy is "constant" over your discretization tile, so that you don't have an issue where sparse discretization might be a problem. $\endgroup$
    – Brale
    Feb 25 '21 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.