2
$\begingroup$

I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for the first 480 steps without training, collecting a replay memory, and then update the parameters each step for the subsequent 240 steps using a window size (of 96 steps) randomly sampled from the replay memory (that always saves the last 480).

My problem is that so far the agent learned the optimal policy just once, and it looks like thisOptimal Q-values (on the test set, training is off) where, as you can see, the agent dynamically changes its evaluation of the state and acts greedily accordingly. All works fine and the performances are optimal, however, I have to slightly change the normalization of the database and rerun the training to get new parameters fitted to the new database.

Trying to get to the same result has been proven impossible so far, (even keeping all settings the same!) because most of the time the agent learns to keep its q-values static such as that. Note: this is an extract of the end of an episode and the beginnig of a new one, the noisy behaviour at the extreme is due to the model being trained for each step, in the middle the training is off and the agent acts greedily (as I explained above). The problem is that the learned parameters give rise to static q values that do not change while the state does and inevitably make the agent stuck in suboptimal strategy, as they never changes actions, even on longer sequences on the test set. The middle part of the second picture, where trainig is off, should look like the first picture, however, I am unable to get back to that optimal behaviour, even keeping all the parameters as they were.

Any idea on what can be the cause of this anomalous behaviour?

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you initializing the agent in a different way each time? I.e a different initial state $\endgroup$
    – hisairnessag3
    Commented Apr 2, 2019 at 15:53
  • $\begingroup$ @hisairnessag3 No, in this case everything is kept the same. Hyperparameters, normalization of the database, optimizer (Nadam). The resulting graphs are obtained after a few hours of training (10-15 at least). I simply cannot reach the same optimal policy in the first place, even keeping all constant, and even less if I change the normalization of the data. $\endgroup$
    – FS93
    Commented Apr 2, 2019 at 16:11
  • $\begingroup$ It seems your weight initialization is likely dynamic in some way then, if not an environment based issue $\endgroup$
    – hisairnessag3
    Commented Apr 2, 2019 at 16:12
  • $\begingroup$ Oh, you meant the weight initialization. Yes, I actually initialize the LSTM layers with he_normal initialization (without keeping track of the seed unfortunately) and the final dense layer with normal distribution. Do you think that this stocasticity can be that detrimental in most cases? I thought that on the long run training would "flatten" any or most initial differences. If not, what do you suggest me to do? $\endgroup$
    – FS93
    Commented Apr 2, 2019 at 16:16

1 Answer 1

Reset to default
1
$\begingroup$

It is likely converging to a far worse local optimum from which it can't recover, so yes I would guess if all else is the same, that is where the issue would be. I would first try to adjust hyperparameters from a static seed for weight initialization. ε and α, are likely good places to start.

If that fails, adjust some architecture params like the number of units or layers in the network as well as the starting seed value.

You seemingly have run into a core problem at the heart of ML research which is reproducibility, which is often predicated by initialization.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks for the suggestion. Let me recap if I got it right. 1- I do not care about reproducibility "per se" but I care in view of having more insights about the dynamics of learning (that I didnt saved), so that for the new runs with the different normalization method, I would have a comparison in terms of hours of training, evolution of q-values, loss etc. So, given the weights issue in order to reproduce the good experiment I would have to run a bunch of new simulations and, hopefully, get there again. But at this point shouldn't I rather start experimenting with the new normalization method? $\endgroup$
    – FS93
    Commented Apr 2, 2019 at 16:43
  • $\begingroup$ 2- About the hyperparameters: should I keep the seed constant and then compare different values of alpha (I dont need epsilon as I can get rewards for every actions at each step), or should I keep alpha (and the other hyperparameters) the same as in the good run and just change different seeds till I get a new optimal result? $\endgroup$
    – FS93
    Commented Apr 2, 2019 at 16:47
  • $\begingroup$ Yea, you can do it either way. I would intuitively just adjust hyperparams as if initialization is an issue, it likely means your model is also not optimal to solve the problem. However, you could just as easily play around with seed values to also reconverge. $\endgroup$
    – hisairnessag3
    Commented Apr 2, 2019 at 16:53
  • $\begingroup$ 3- A bit off-topic: I normalized the database by dividing it into blocks of 96 steps and normalizing each block (independently) with its own Zscore. This is good for testing on another database but not viable for an online real application, where I get one element of the timeseries at the time, thus I have to change the normalization in a way that each element is normalized only according to its past data points. I am afraid that this might be detrimental to the performances as I would not be "leaking" any bit of information about the future. Do you think this can be a big issue? $\endgroup$
    – FS93
    Commented Apr 2, 2019 at 16:53
  • $\begingroup$ For your 3rd point are you only doing this via experience replay or in the normal training loop? $\endgroup$
    – hisairnessag3
    Commented Apr 2, 2019 at 16:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .