I am currently reading and trying to understand the theory behind TRPO, i.e. sections 2 and 3 from the paper here.

Ultimately, I want to apply PPO to do a (single) stock trading task using the FinRL library, see here. From what I have read so far, TRPO is kind of a preliminary stage to PPO, so I wanted to get a good understanding of TRPO (Is this a good idea?).

I just recently started reading into RL, so I have no practical experience and I was wondering if and how TRPO could be applied in my case. My state space should have a form like this $$ \mathcal{S} := \mathbb{R}_+ \times \mathbb{Z}_+ \times \mathbb{R}_+ \times \mathbb{R}^n, $$ where $\mathbb{R}_+$ indicates my bank account, $\mathbb{Z}_+$ the number of stocks I own, $\mathbb{R}_+$ the closing price of the stock at a given time and $\mathbb{R}^n$ stands for a number $n$ of different indicators. My action space would be $\mathcal{A} := \{-k,\dots,0,\dots,k\}$, where $k$ is the maximum number of shares I can buy.

This state space is of course not finite. I wanted to see if TRPO would work the same from a theoretical viewpoint and already started to redo the entire proof of the theory given in the TRPO paper not restricting myself to a finite state space and from what I found so far, it would be quite the same with some integrals instead of sums.

Regardless, is this the right way? Is it worth to continue the proof, or would you do some sort of discretisation in this case in practice, or would you not apply TRPO to continuous state spaces in the first place?



You must log in to answer this question.

Browse other questions tagged .