2
$\begingroup$

In the literature that I've seen so far on how to either exactly or approximately solve POMDPs (Partially-Observable Markov Decision Processes), there seems to be a lot of focus placed on maintaining a distribution over all possible states of what the current state is as well as approximating the value function as a function of this distribution at each step.

Is this level of rigour how things are implemented in practice?

For example, say we have a market with cash and a single other asset that our agent participates in.

First, our agent receives an observation: $o_0 = (t_0, b_0, h_0)$ where $t_i$ denotes the time of the observation, $b_i$ denotes some actionable observation of the market such as current bid/ask price of the asset, and $h_i$ denotes our holdings of cash and the other asset.

Then, our agent chooses an action $a_0 \in \mathcal{A}_0$ which is to either sell or buy a specified amount of the asset or do nothing.

Then our agent receives the next observation $o_1 = (t_1, b_1, h_1)$ and also receives a reward, $r_1 = L(h_1, b_1) - L(h_0, b_0)$ where $L(h_i, b_i)$ denotes the value of the portfolio with holdings, $h_i$, as determined by the market representation $b_i$

This is then repeated for the remainder of the potentially-infinite horizon of the POMDP.

If our goal is to maximimize the sum of the rewards over this potentially-infinite horizon and we are able to interact with this environment "live" (as in we are not simulating the dynamics, our agent is directly interacting with the market), how would we go about approaching this in practice if it does differ from the theoretical approaches?

$\endgroup$

1 Answer 1

2
$\begingroup$

A formal POMDP approach that models a belief state (a distribution over possible states) needs to have a theoretical model for the hidden aspects of the state. This is not always appropriate or possible.

If you have no theoretical MDP that underlies the POMDP (e.g. possible sets of cards an opponent holds in poker), and are limited in not only not knowing hidden data, but also not knowing what structure hidden data might have or the rules governing it, then you need to look at other options.

One approach is to try to extract maximum information that the full observation history can provide. This being the only data that you have, is the best you can do if you lack a theoretical model. This full history can include all observed state data, actions and rewards.

If the data can be summarised logically into a fixed size - perhaps with some theoretical justification in the problem domain - you can use that. If not, you can attempt to learn an abstract summary from experience by making your value function, or your policy function, or a separate "helper" function (e.g. that predicts next observables) into a sequence-based predictor such as an LSTM.

Predicting market behaviour and making investment choices are not known to be easy targets for machine learning models. This is in part because all the low hanging fruit for statistical analysis has been researched and exploited for many years already. There are skilled mathematicians who work in this area professionally, who already understand how to create sophisticated predictive models, and they are your competition for any agent that you could write. It is very hard to create an autonomous agent that can do well in this kind of environment.

$\endgroup$
3
  • $\begingroup$ Thank you so much for this detailed and insightful response. However, I have a few questions: 1. What do you mean by "summarised logically into a fixed size"? 2. Would you be able to provide a very high level example of the "attempt to learn an abstract summary..." perhaps as applied to this financial example? 3. How do we predict future observables if they depend on our actions taken? do we just add this into our model? Thanks again :) $\endgroup$
    – QMath
    Feb 21 at 20:07
  • 1
    $\begingroup$ @QMath I think trying to answer those follow-on questions here would make the answer too long. Apart from 3 - yes you should add action history and current action. Action history may easily affect hidden state, and current action very likely to impact next observables. For other follow up questions, they would be good new questions on the site $\endgroup$ Feb 21 at 21:57
  • $\begingroup$ That's fair regarding the first two questions and great, that makes sense regarding the third. Thanks again! $\endgroup$
    – QMath
    Feb 21 at 22:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .