# Understanding "belief states" in Bayesian RL

In Bayesian RL, the uncertainty about the transition probabilities $$\mathcal{P}$$ parameterized by $$\theta$$ is captured by viewing $$\theta$$ as a random variable, and updating the pdf of $$\theta$$ after observing data, as shown in the slide below taken from here.

Outside of a RL setting, (e.g. in any elementary statistics course), we can generically capture uncertainty about a distribution's parameters by assuming the parameters themselves are random variables. Therefore it seems sufficient to continuously update the parameter $$\theta$$ as observations roll in.

However, in the formulation in the slide above, and in some literature, we see the set $$B$$ being described a set of belief states, and $$b \in B$$ being essentially defined as a function acting on $$\theta$$.

What I am confused about is this notion of treating $$b$$ as a state. For example, in the slide below, assuming a finite and discrete setting, we can completely replace every $$b(\theta)$$ with $$Pr(\theta)$$, we can replace each $$b$$ in the conditional as $$\theta = x$$, where $$x$$ is just some value, and the meaning would be the same.

Question 1: What is the purpose of viewing the parameter updating process as states?

Question 2: What is an arbitrary element of $$b \in B$$? In the slides here, $$b(\theta)$$ simply gives you the probability of $$\theta$$ equaling some value. Why would we need a set $$B$$ to do this?

Question 3: Suppose there are $$|\mathcal{S}|$$ states and $$|\mathcal{A}|$$ actions, both of which are finite. Assume a tabular setting. What is the cardinality of $$B$$? Is it $$|\mathcal{S}|^2|\mathcal{A}|$$ belief states?

• Please, next time, ask only one question per post, even though the questions are related. Why? For several reasons. 1. Readers may only know the answer to a subset of the questions. 2. You simplify the life of the readers which can focus on one question, 3. in the future, it's unlikely that other people have exactly this same combination of questions.
– nbro
Jun 4 at 14:37

Question 2: What is an arbitrary element of $$b \in B$$?

Most generally, an arbitrary element of $$B$$ is a probability function over the parameter space $$\Theta$$:

$$b \in B,\;\theta\in\Theta$$ $$b:\Theta\to[0,1],\quad \text{with}\quad \int\limits_{\theta\in\theta} b(\theta)d\theta = 1$$

The parameter space $$\Theta$$ is not formally mentioned on the slides. But, it seems, that they consider a particular case when $$\Theta=[0,1]$$ interval (since they use beta functions over it).

Question 3: What is the cardinality of $$B$$?

The cardinality of $$B$$ doesn't depend on $$|\mathcal{S}|$$ or $$|\mathcal{A}|$$. In the most general case, the cardinality of $$B$$ is the continuum of all functions $$B$$ over the parameter space $$\Theta$$. That's what makes the general treatment of Bayesian RL and POMDPs so difficult.

The slides constrain themselves to beta distributions - which is a class of probability distributions, typically used to deal with uncertain probabilities.

Question 1: What is the purpose of viewing the parameter updating process as states?

The "Bayesian RL" approach, discussed on the slides, looks very similar to the framework of Partially observable Markov decision processes (POMDPs). There are differences between the two approaches, but the general idea is the same - making decisions under imperfect information.

The "basic" MDP framework assumes that an agent have perfect knowledge about the current state $$s$$ and that we know the exact dynamics in the form of transition probabilities $$P(s'|s,a)$$. These assumptions are too strong for many practical applications. So the basic MDP approach gets augmented with ability to deal about probability distribution over states and/or dynamics.