I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state space where the state is a belief (distribution) with some unknown parameters.

Are they not the same thing?

  • $\begingroup$ Hi. Maybe, for completeness, you should also explain where you heard the term "Bayes-adaptive MDP". $\endgroup$
    – nbro
    Jul 27, 2020 at 13:14

1 Answer 1


Both Belief-MDPs and Bayes-Adaptive MDPs (BAMDPs) are special cases of POMDPs and their state space is augmented with a belief over their unobserved/hidden variables.

In a belief-MDP, the hidden variables can change over the course of an episode. (Eg. Both the position and the uncertainty in the position of the robot can vary during an episode).

In a BAMDP, the hidden variables are usually the attributes of the transition/reward function and are held constant during an episode. (Eg. In a robot locomotion task, ground friction or load - dynamics attributes, current goal location - reward function attributes. Though variables defining these attributes are unknown to the agent and the agent has to infer them, the actual variables remain unchanged during an episode)


  1. M Ghavamzadeh, S Mannor, J Pineau, Aviv Tamar - Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359-483, 2015.

  2. L Zintgraf, K Shiarlis, M Igl, S Schulze, Y Gal, K Hofmann, and S Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .