0
$\begingroup$

Let's say an RL trading system places trades based on pricing data.

Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what is the most action with the highest utility.

The state is a sequence of prices, and the action is either buy, hold, sell.

Instead of "Loop for each episode" as per the Sarsa algorithm :

enter image description here

I add an additional outer loop. Now instead of just looping for each episode we have:

for 1 to N:     
   "Loop for each episode"

Manually set N or exit out of the loop on convergence.

Is this the correct approach? Iterating multiple times over the episodes will produce more valuable state-action pairs in the Q table, because e greedy is not deterministic and for each iteration may exploit an action to greater reward than other episode epochs.

Improve this question
$\endgroup$
7
  • $\begingroup$ do you have a fixed data set or are you able to generate data from the environment? if it is the latter, then yes your approach is correct $\endgroup$
    – David
    Commented Jun 18, 2020 at 15:14
  • $\begingroup$ @David Ireland the dataset is fixed as for a given time interval the price values do not change. I'm confused by your comment 'generate data' as the model utilises data from the environment, the source of the environment pricing data is a trading exchange. So I can generate data by reading the pricing data but I'm not sure if I'm interpreting correctly 'generate data from the environment' . $\endgroup$
    – blue-sky
    Commented Jun 18, 2020 at 15:29
  • $\begingroup$ The way I understood your post was that you have an episode that is 24 steps long. Each step you get a series of prices and you have to decide whether to hold/sell/buy. If so, then are you able to interact with an environment arbitrarily, e.g. can you run simulations of this? or is your data set fixed? $\endgroup$
    – David
    Commented Jun 18, 2020 at 16:58
  • $\begingroup$ @DavidIreland yes can run simulations where the reward of choosing buy/hold/sell for each step is observed. The dataset is fixed in that the number of prices for the environment is immutable. The episode length is mutable, for example If I set the episode length from 1 to two hours then there are 12 episodes available and the number of price attributes for each state will double. $\endgroup$
    – blue-sky
    Commented Jun 18, 2020 at 17:45
  • $\begingroup$ Okay, then it seems like it should work by following the table. However, is your state space continuous? If so, then you may want to use Deep Q-Learning. $\endgroup$
    – David
    Commented Jun 18, 2020 at 18:32

1 Answer 1

Reset to default
1
$\begingroup$

Given that your state space is continuous, then I would recommend using Deep Q-Learning. As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .