Timeline for What is the intuition behind grid-based solutions to POMDPs?
Current License: CC BY-SA 4.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 5, 2020 at 23:30 | comment | added | nbro | Now, if we wanted to find $v(b_2)$, we could SOMEHOW combine $v(b_1)$ and $v(b_3)$, i.e. $v(b_2) = g(v(b_1), v(b_3))$, where $g$ is some function that combines the inputs. Note that I don't know if this is actually done in practice. I am just giving you the GENERAL idea behind interpolation!! | |
| Apr 5, 2020 at 23:27 | comment | added | nbro | In our context, we have the value of the belief states at certain grid points (we compute this with e.g. one value iteration algorithm for POMDPs). If we wanted to find the value of the belief states not at the grid points, we would compute them SOMEHOW by using the value of the belief states at the grid points (which we have!). You don't introduce any new action or whatever. To be more concrete, assume that we have the value at the belief states $b_1, b_3, b_5$ (which correspond to some grid points), i.e. we have $v(b_1)$, $v(b_3)$ and $v(b_5)$ (computed with value iteration). | |
| Apr 5, 2020 at 23:23 | comment | added | nbro | @FourierFlux Then you say "Do you define new actions", but I don't understand why you would define new actions. Then you say "Doesn't interpolation just lead back to the same issue of having a continuous state space?", interpolation is an operation that allows you to find the values of a function at points where you don't have those values. For instance, if you have $f(a)$ and $f(b)$ and you want to find the value of $f$ at $a \leq c \leq b$, then you can SOMEHOW combine $f(a)$ and $f(b)$ to get $f(c)$. | |
| Apr 5, 2020 at 23:21 | comment | added | nbro | @FourierFlux I think your first question "Lets consider a MDP with a continuous state space, discretizing it still leaves the problem of your actions giving states not in the discretization" is related to the discretization of continuous space MDPs and how it works. Maybe you could ask a new question. You're very confused. First of all, do you understand that if you find the value function you can derive the policy from it? So, if that's the case, we just find a value function in order to solve an MDP. How? There is a value iteration algorithm for MDPs and there's also one for POMDPs. | |
| Apr 5, 2020 at 22:47 | comment | added | FourierFlux | Lets consider a MDP with a continuous state space, discretizing it still leaves the problem of your actions giving states not in the discretization, where exactly does the interpolation come in when it comes to finding a policy and how does it help? Do you define new actions and round the resulting state to the "closest" one? Doesn't interpolation just lead back to the same issue of having a continuous state space? | |
| Apr 5, 2020 at 20:14 | comment | added | nbro | When you say "an action can take a point in the belief space grid and update it to no longer be in the grid". Well, it's true that an action may move the environment to a state that is not represented in the grid. That's why you may need to use something like interpolation. Are you familiar with the concept of interpolation? | |
| Apr 5, 2020 at 20:04 | comment | added | FourierFlux | Thanks, I will read them. I think my confusion is that the belief state is continuous and an action can take a point in the belief space grid and update it to no longer be in the grid. So it's not clear to me how this is addressed when it comes to computations. Even restricting your actions to a smaller subset won't mean they will always result in transforming a grid point to a grid point. When transforming the POMDP into a discritized MDP, what defines your set of actions and associated probabilities? | |
| Apr 5, 2020 at 15:06 | history | edited | nbro | CC BY-SA 4.0 |
added 104 characters in body
|
| Apr 5, 2020 at 14:59 | history | edited | nbro | CC BY-SA 4.0 |
added 104 characters in body
|
| Apr 5, 2020 at 14:52 | history | edited | nbro | CC BY-SA 4.0 |
deleted 145 characters in body
|
| Apr 5, 2020 at 14:47 | history | edited | nbro | CC BY-SA 4.0 |
deleted 145 characters in body
|
| Apr 5, 2020 at 14:42 | history | edited | nbro | CC BY-SA 4.0 |
deleted 145 characters in body
|
| Apr 5, 2020 at 14:35 | history | edited | nbro | CC BY-SA 4.0 |
deleted 145 characters in body
|
| Apr 5, 2020 at 14:21 | history | edited | nbro | CC BY-SA 4.0 |
added 294 characters in body
|
| Apr 5, 2020 at 13:51 | comment | added | nbro | @FourierFlux I don't understand what you mean by "outside of the state space" in "basically your state can be transformed to something outside your state space". Anyway, I've updated this answer to include more details and references. | |
| Apr 5, 2020 at 13:49 | history | edited | nbro | CC BY-SA 4.0 |
deleted 25 characters in body
|
| Apr 5, 2020 at 4:51 | comment | added | FourierFlux | Thanks, I am still confused on this approximation method though. Actions have associated probability distributions that produce belief states which may not be on the grid, basically your state can be transformed to something outside your state space, so how do you handle this? | |
| Apr 5, 2020 at 3:28 | history | edited | nbro | CC BY-SA 4.0 |
added 140 characters in body
|
| Apr 5, 2020 at 3:23 | history | answered | nbro | CC BY-SA 4.0 |