Model-based RL for time series data

Question

I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is.

To be concrete: I have $X(t)$ and $a(t-1)$, where $X(t)$ is n-dimensional time-series data and $a(t)$ is 1-dimensional time-series data. At time $t$, they together represent the observation/state space. Also, at time $t$, the agent makes a decision about $a(t)$. This decision (action) $a(t)$ directly defines the next state space $X(t+1)$ and rewards, by some function $f$ & $g$, $f(a(t), X(t)) = X(t+1)$ and $g(a(t), X(t)) = R(t+1)$.

I have to estimate this impact, i.e. where I will end up (what will be the next state). I decided to use a model-based RL algorithm, because, from my knowledge, model-based RL does exactly this.

Can you advise me on a good paper and Github code, to implement this project?

As I noticed, there do not exist many works on Model-based RL.

Answer 1

To my knowledge, there does not exist anything along the lines of model-based reinforcement learning with time-sensitive data. I think the best chance you have is to try to abstract the data that you have into a model which is not time-sensitive.

What would happen when you get past the timestamps of your original data? I am guessing that when testing this model, you will have states with time stamps past your original latest timestamp. Or are you only using your model in the time stamps of your training data? Is it going to be a useful model then?

Firstly, please note that the topics of this forum are mostly theoretical. This question leans on the boundary. If you frame your question a bit differently, ending your post with a clear question and also putting that question in the title, you will get more responses.

I feel like the question is 'What are the possibilities for model-based reinforcement learning for time-sensitive data?' But this question is inherently vague as it implies that you have a model according to the definitions of models in RL, which are not time sensitive. If you would like a better answer then please provide more information or ask your question on a more practical forum if it is more practical than what I guessed.

Disclaimer: would have just put some comments under the post as this is not a real sufficient answer, but my reputation is not high enough for that :P

Answer 2

It seems that your description perfectly matches the naive RL approach. What you can do with model-based RL is perform rollouts with the model to predict future states. In other words, with an accurate model you might predict the next state X(t + 1) given the current state X(t) and applying some action a(t). The following (and recently) paper shows how these rollouts works.