1
$\begingroup$

I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state space where the state is a belief (distribution) with some unknown parameters.

Are they not the same thing?

$\endgroup$
1
  • $\begingroup$ Hi. Maybe, for completeness, you should also explain where you heard the term "Bayes-adaptive MDP". $\endgroup$ – nbro Jul 27 '20 at 13:14
2
$\begingroup$

Both Belief-MDPs and Bayes-Adaptive MDPs (BAMDPs) are special cases of POMDPs and their state space is augmented with a belief over their unobserved/hidden variables.

In a belief-MDP, the hidden variables can change over the course of an episode. (Eg. Both the position and the uncertainty in the position of the robot can vary during an episode).

In a BAMDP, the hidden variables are usually the attributes of the transition/reward function and are held constant during an episode. (Eg. In a robot locomotion task, ground friction or load - dynamics attributes, current goal location - reward function attributes. Though variables defining these attributes are unknown to the agent and the agent has to infer them, the actual variables remain unchanged during an episode)

References:

  1. M Ghavamzadeh, S Mannor, J Pineau, Aviv Tamar - Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359-483, 2015.

  2. L Zintgraf, K Shiarlis, M Igl, S Schulze, Y Gal, K Hofmann, and S Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.