1
$\begingroup$

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results on testing data (portfolio change at the end of test period). I think, its happening due random initialisation of NN parameters and solution reaching different local maxima. So, how am I to evaluate the agent if it gives anywhere from negative to positive change on every train?

$\endgroup$
1
  • 1
    $\begingroup$ Is the trading bot trained only on price data? If so, there is typically no usable pattern in that, and you should expect the bot to do about as well as throwing darts at a board with "buy" and "sell" written on it, even if your training was successful. Although there are other possibilities, including that something is wrong with your implementation or training approach or hyperparameters. $\endgroup$ Jul 26, 2023 at 17:30

2 Answers 2

0
$\begingroup$

I suggest you should try fixing the seeds. Here's the code how to do it, if you are using torch and the environment is inherited from gym

import torch
import random
import numpy as np

env = import gymnasium as gym
env = gym.make("LunarLander-v2")

def set_global_seed(env, seed=42):
    random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True
    env.seed(seed)
    env.action_space.seed(seed)

set_global_seed(env)
$\endgroup$
4
  • $\begingroup$ In that case should I experiment with various seeds? In some train session I got good results. Or should the whole process be agnostic of seed i.e. set the seed once and not revisit seed setting ever again. I'm asking this because what if the seed leads to a poor testing result. $\endgroup$
    – ad124j2
    Feb 26, 2023 at 16:44
  • $\begingroup$ The seed will always affect the result. This paper experiments with 10^4 seeds and finds it true. However, I recommend you to select about 5 seeds, e.g., 0, 1, 2, 3, 4, then tune the hyperparameters based on those seeds and never touch again. You can do 10 if you are unsure about it $\endgroup$ Feb 26, 2023 at 16:55
  • $\begingroup$ What if I want to evaluate my architecture? I was thinking of training the agent multiple times and calculate average change which will give me a good idea of expected value of the change. What do you think of this approach? $\endgroup$
    – ad124j2
    Feb 26, 2023 at 17:04
  • $\begingroup$ That would be the same thing as I mentioned earlier. Fix the seed, do anything you want $\endgroup$ Feb 26, 2023 at 17:24
0
$\begingroup$

The problem is not that the algorithm is getting inconsistent results and you need to nail it down to one of the "good" strategies. It is instead that the algorithm is getting inconsistent results and you need to figure out how to get consistently good results. Depending on how you've tested the good strategy the model could just be over or under-fit and will perform poorly in production.

If there is a good solution, on a long enough timescale value estimation should converge, no matter the initialization. In supervised learning we turn to cross validation to give some indication that a model will do well on unseen data. It is not enough to pick a well behaving initialization there and the principle holds in RL as well. DQN was not impressive because the authors found a single initialization that played pong, they found an algorithm and data representation that could play pong.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .