Questions tagged [pomdp]

For questions related to the concept of Partially Observable Markov Decision Process (POMDP), which is a generalization of the Markov Decision Process (MDP) to the cases where information about the states is incomplete (or partially observable).

Filter by
Sorted by
Tagged with
2 votes
1 answer
62 views

How are POMDPs solved in practice?

In the literature that I've seen so far on how to either exactly or approximately solve POMDPs (Partially-Observable Markov Decision Processes), there seems to be a lot of focus placed on maintaining ...
QMath's user avatar
  • 123
1 vote
0 answers
17 views

Control algorithms when system dynamics are stochastic and/or unknown

I'm working on a traffic signal control problem, which I am currently approaching with Reinforcement Learning, but I want to try some other control algorithms. This is hard for me because we don't ...
Federico Taschin's user avatar
0 votes
0 answers
23 views

What methods are available for this kind of RL with partially unknown rewards?

Let me give an example. There is a king with 1 million subjects. He wants to maximize the discounted sum of future happiness of these subjects. However, he never fully knows their happiness. He can ...
causative's user avatar
  • 149
2 votes
1 answer
316 views

What's the relationship between Bayesian RL and POMDPs?

Bayesian RL seems concerned with having uncertainty over the transition function of the environment, while POMDPs try to capture uncertainty over the state one is currently in. However, both end up ...
mdc's user avatar
  • 380
4 votes
1 answer
363 views

Are multi agent or self-play environments always automatically POMDPs?

As part of my thesis, I'm working on a zero sum game with RL to train an agent. The game is a real-time game, a derivation of pong, one could imagine playing pong with both sides being foosball rods. ...
kitaird's user avatar
  • 115
13 votes
2 answers
2k views

Is there a fundamental difference between an environment being stochastic and being partially observable?

In AI literature, deterministic vs stochastic and being fully-observable vs partially observable are usually considered two distinct properties of the environment. I'm confused about this because what ...
martinkunev's user avatar
2 votes
1 answer
113 views

Is there a mathematical formalism to deal with a missing reward signal?

Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get ...
Onil90's user avatar
  • 183
0 votes
1 answer
177 views

Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?

Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o_t$) with a camera at each location (action $...
Schlozma's user avatar
6 votes
3 answers
3k views

What exactly are partially observable environments?

I have trouble understanding the meaning of partially observable environments. Here's my doubt. According to what I understand, the state of the environment is what precisely determines the next state ...
CHANDRASEKHAR HETHA HAVYA's user avatar
2 votes
1 answer
65 views

Is there a way of path reconstruction using only the history of belief states?

Given a history of belief states, is there a common method that backtracks the most likely path of ending up in the current belief state? I have a Markov model which calculates belief states after ...
MScott's user avatar
  • 445
2 votes
0 answers
89 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
O'Jhene's user avatar
  • 75
1 vote
0 answers
92 views

How do I learn the value function for a POMDP with a single-step horizon (bandit)?

Consider a POMDP with a finite number of environment states, $|\mathcal{S}| = N$, but the number of belief states is uncountably infinite. The belief state space is the convex hull of an $N$ simplex. ...
jdizzle's user avatar
  • 61
4 votes
0 answers
149 views

How to update the observation probabilities in a POMDP?

How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model? The POMDP relies on observation probabilities that match an observation to a state....
Pluxyy's user avatar
  • 85
1 vote
0 answers
19 views

How to obtain the policy in the form of a finite-state controller from the value function vectors over the belief space of the POMDP?

I was reading this paper by Hansen. It says the following: A correspondence between vectors and one-step policy choices plays an important role in this interpretation of a policy. Each vector in $\...
Rnj's user avatar
  • 221
1 vote
0 answers
32 views

Understanding example for Improved Policy Iteration for POMDPs

I was going through this paper by Hansen. This paper proposes policy improvement by first converting set of $\alpha$ vectors into finite state controller and then comparing them to obtain improved ...
Rnj's user avatar
  • 221
2 votes
0 answers
367 views

How does one stack multiple observations in the input layer of a convolutional neural network?

The paper, Deep Recurrent Q-Learning for Partially Observable MDPs, talks about stacking multiple observations in the input of a convolutional neural network. How does this exactly work? Do the ...
desert_ranger's user avatar
4 votes
1 answer
2k views

Why is tic-tac-toe considered a non-deterministic environment?

I have been reading about deterministic and stochastic environments, when I came up with an article that states that tic-tac-toe is a non-deterministic environment. But why is that? An action will ...
EEAH's user avatar
  • 193
1 vote
1 answer
806 views

What is the difference between Bayes-adaptive MDP and a Belief-MDP in Reinforcement Learning?

I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state ...
gauzah's user avatar
  • 11
2 votes
0 answers
79 views

If the performance of an RL agent in a partially observable environment is "good", is this likely only accidental?

In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper ...
unter_983's user avatar
  • 331
1 vote
2 answers
275 views

Does "transition model" alone in an MDP imply it's non-deterministic?

I am looking at a lecture on POMDP, and the context is that, when the quadcopter can't see the landmarks, it has to use reckoning. And then he mentions the transition model is not deterministic, hence ...
gfdsal's user avatar
  • 171
1 vote
0 answers
152 views

How to choose an RL algorithm for a Gridworld that models a much more complex problem

I am considering using Reinforcement Learning to do optimal control of a complex process that is controlled by two parameters $(n_O, n_I), \quad n_I = 1,2,3,\dots, M_I, n_O = 1,2,3,\dots, M_O$ In this ...
tmaric's user avatar
  • 382
3 votes
0 answers
93 views

Can we use a Gaussian process to approximate the belief distribution at every instant in a POMDP?

Suppose $x_{t+1} \sim \mathbb{P}(\cdot | x_t, a_t)$ denotes the state transition dynamics in a reinforcement learning (RL) problem. Let $y_{t+1} = \mathbb{P}(\cdot | x_{t+1})$ denote the noisy ...
math_phile's user avatar
3 votes
0 answers
193 views

Is Monte Carlo tree search needed in partially observable environments during gameplay?

I understand that with a fully observable environment (chess / go etc) you can run an MCTS with an optimal policy network for future planning purposes. This will allow you to pick actions for gameplay,...
Yohahn Ribeiro's user avatar
3 votes
1 answer
253 views

What is the intuition behind grid-based solutions to POMDPs?

After spending some time reading about POMDP, I'm still having a hard time understanding how grid-based solutions work. I understand the finite horizon brute-force solution, where you have your ...
FourierFlux's user avatar
1 vote
0 answers
156 views

Why can the core reinforcement learning algorithms be applied to POMDPs?

Why can an AI, like AlphaStar, work in StarCraft, although the environment is only partially observable? As far as I know, there are no theoretical results on RL in the POMDP environment, but it ...
FourierFlux's user avatar
6 votes
0 answers
187 views

How exactly does self-play work, and how does it relate to MCTS?

I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
Alienator's user avatar
2 votes
0 answers
97 views

What are some approaches to estimate the transition and observation probabilities in POMDP?

What are some common approaches to estimate the transition or observation probabilities, when the probabilities are not exactly known? When realizing a POMDP model, the state model needs additional ...
MScott's user avatar
  • 445
2 votes
0 answers
41 views

Finding total number of states in a POMDP

I've been working on a question that is posed in a document I've been reading, that models qualifying for a job as a POMDP. In this model, a person takes 3 exams, and must pass all of them in order to ...
zay's user avatar
  • 21
6 votes
2 answers
723 views

Are perfect and imperfect information games modelled as fully and partially observable environments, respectively?

In perfect information games, the agent can see all the moves performed in the past. Besides, it can observe the next action that will be put into practice by the opponent. In this case, can we say ...
Goktug's user avatar
  • 161
3 votes
1 answer
427 views

Is it possible for value-based methods to learn stochastic policies?

Is it possible for value-based methods to learn stochastic policies? I'm trying to get a clear picture of the different categories for RL algorithms, and while doing so I started to think about ...
Krrrl's user avatar
  • 211
4 votes
0 answers
364 views

Is there a way to do reinforcement learning in POMDP?

Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed? More ...
Deepanshu Vasal's user avatar
10 votes
1 answer
5k views

Can Q-learning be used in a POMDP?

Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial ...
drerD's user avatar
  • 298
2 votes
1 answer
155 views

Can the normalization factor for the belief state update be zero?

In order to update the belief state in a POMDP, the following formula is used: $$b'(s')=\frac{O(a, s', z) \sum_{s\in S} b(s)T(s, a, s')}{\mathbb{P}(z \mid b, a)}$$ where $s$ is a specific state in ...
Bryan McGill's user avatar
2 votes
1 answer
157 views

Does the observation function for POMDP always add up to 1?

I was reading in the article A tutorial on partially observable Markov decision processes (p. 120), by Michael L. Littman, that $\sum_{z \in Z}O(a, s',z) =1$, where $a$ is the action, $s'$ the next ...
Bryan McGill's user avatar
2 votes
1 answer
2k views

Is my understanding of the differences between MDP, Semi MDP and POMDP correct?

I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature ...
Rui Nian's user avatar
  • 433
2 votes
2 answers
312 views

How to define a reward function in POMDPs?

How do I define a reward function for my POMDP model? In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...
Bryan McGill's user avatar
5 votes
1 answer
641 views

What could happen if we wrongly assume that the POMDP is an MDP?

Consider the Breakout environment. We know that the underlying world behaves like an MDP, because, for the evolution of the system, it just needs to know what the current state (i.e. position, speed, ...
Marco Favorito's user avatar