Bayesian RL seems concerned with having uncertainty over the transition function of the environment, while POMDPs try to capture uncertainty over the state one is currently in. However, both end up reducing to something like a belief MDP: an MDP in which everything is known, but the states are points in belief space.
Could anyone explain what the differences between these are in more detail?
After looking more into this, I think I have a better understanding.
In Bayesian RL, one has uncertainty over the transition function of the environment. For example, we might know that our robot can slip when transitioning state, but we don't know what the probability of slip $\theta$ is (although we have some prior over it).
In POMDPs, one has uncertainty over the current state. For example, we might not know exactly where the robot is located, and we try to understand that from observations (which are a function of the state, e.g. sensor readings).
In both POMDPs and Bayesian RL, a similar trick applies: one can "tuck one's uncertainty into the state", reducing both problems to MDPs. This is nice because we already have a bunch of methods developed to solve MDPs, so we don't have to reinvent the wheel. However, these are a particular type of MDP: in which each state corresponds to a specific "belief-state".
How the transition dynamics are computed in POMDPs and Bayesian RL are obtained is also subtly different, but the idea is similar: one simply incorporates the latest information one gains through observations, e.g. "did the robot slip this last timestep?", or "what were my last sensor readings this last timestep?".
For more information on POMDPs, see "Planning and acting in partially observable stochastic domains" by Kaelbling. For more info on Bayesian RL, this lecture seems like a good starting point.
