After looking more into this, I think I have a better understanding.
In Bayesian RL, one has uncertainty over the transition function of the environment. For example, we might know that our robot can slip when transitioning state, but we don't know what the probability of slip $\theta$ is (although we have some prior over it).
In POMDPs, one has uncertainty over the current state. For example, we might not know exactly where the robot is located, and we try to understand that from observations (which are a function of the state, e.g. sensor readings).
In both POMDPs and Bayesian RL, a similar trick applies: one can "tuck one's uncertainty into the state", reducing both problems to MDPs. This is nice because we already have a bunch of methods developed to solve MDPs, so we don't have to reinvent the wheel. However, these are a particular type of MDP: in which each state corresponds to a specific "belief-state".
- In POMDPs, each belief-state will only contain a belief over which state one is (e.g. a distribution over x-y locations).
- In Bayesian RL, each belief-state will contain both a state and a belief over the unknown parameters of the transition dynamics (e.g. a Beta distribution over $\theta$).
How the transition dynamics are computed in POMDPs and Bayesian RL are obtained is also subtly different, but the idea is similar: one simply incorporates the latest information one gains through observations, e.g. "did the robot slip this last timestep?", or "what were my last sensor readings this last timestep?".
For more information on POMDPs, see "Planning and acting in partially observable stochastic domains" by Kaelbling. For more info on Bayesian RL, this lecture seems like a good starting point.