Suppose $x_{t+1} \sim \mathbb{P}(\cdot | x_t, a_t)$ denotes the state transition dynamics in a reinforcement learning (RL) problem. Let $y_{t+1} = \mathbb{P}(\cdot | x_{t+1})$ denote the noisy observation or the imperfect state information. Let $H_{t}$ denote the history of actions and observations $H_{t+1} = \{b_0,y_0,a_0,\cdots,y_{t+1}\}$.

For the RL Partially Observed Markov Decision Process (RL-POMDP), the summary of the history is contained in the "belief state" $b_{t+1}(i) = \mathbb{P}(x_{t+1} = i | H_{t+1})$, which is the posterior distribution over the states conditioned on the history.

Now, suppose the model is NOT known. Clearly, the belief state can't be computed.

Can we use a Gaussian Process (GP) to approximate the belief distribution $b_{t}$ at every instant $t$?

Can Variational GP be adapted to such a situation? Can universal approximation property of GP be invoked here?

Are there such results in the literature?

Any references and insights into this problem would be much appreciated.

  • 1
    $\begingroup$ How? If you have no model you literally know nothing. You have no idea how to assign a probability to any state or even what a state is. $\endgroup$ – FourierFlux May 12 '20 at 9:58
  • $\begingroup$ @FourierFlux: Say you have access to the following (training) data: Input {a_t,y_t} and output {x_t}. It's just during the implementation (testing), we only have access to imperfect information. Could a GP, which captures the distribution over the states, match the belief state? at every time t? Are there such works? $\endgroup$ – math_phile May 12 '20 at 15:09
  • $\begingroup$ @math_phile You need a model for that. GP is a model. You said you have no model. You need to make some assumptions about what you want to predict. $\endgroup$ – TMS Jan 13 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.