2
$\begingroup$

I am playing around with a DRL agent in a stock-trading environment.

I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't come from the environment?

For example, I have included things like "current account balance" and "current unrealized gain" in my observation space (as I believe it's useful). However, I don't know how I could normalize these values, given that they are dependent on what actions the agent took, which changes every time etc.

Any feedback or advice is appreciated.

Will it be detrimental if I don't normalize these values (as long as they're reasonably within the orders of magnitude of my other normalized variables)?

I guess a simple example would be like if a robot was being trained to pick up balls, and one of the observations was "current number of balls picked up", how would you normalize that value, given that it's just a count that could technically go to infinity?

$\endgroup$

3 Answers 3

2
+50
$\begingroup$

The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization will be dependent on the actions the agent takes; this is not unique to your problem.

As for your concern of the state observations going to infinity, this will not happen. Keeping with your example, the number of balls going to infinity would require the number of timesteps to also go to infinity. In practice there will have to be some finite length of the episodes, so this won't be an issue. And note, once you start a new episode, the number of balls goes back to 0, so it's not like the normalization amount can go to infinity either.

$\endgroup$
4
  • $\begingroup$ Thank you for your response. I had considered a running mean/StdD, but my concern is that unless I have really long episodes (which I don't), there isn't enough time for these values to reach any kind of stability. What do you think? $\endgroup$ Jun 12 at 15:11
  • $\begingroup$ The length of the episode should not be important here. After you have hundreds of thousands (or millions) of transitions, the change to the mean/stdev from a single episode (which consists of perhaps a few hundred or ~thousand transitions) is fairly small. $\endgroup$
    – Taw
    Jun 12 at 16:15
  • $\begingroup$ So, you're saying I would normalize across episodes? Not just within 1 episode? So, episode 1 ends, but the state variables of episode 2 are normalized by the mean/SD of episode 1 in addition to the timesteps taken so far in episode 2? $\endgroup$ Jun 12 at 16:19
  • $\begingroup$ yes. the normalization amount should be the empirical mean/stdev, computed over EVERY observed transition. $\endgroup$
    – Taw
    Jun 12 at 16:28
2
$\begingroup$

It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that.

But it might cause problems later, if the values move outside the interval they were trained on. For example, it might work perfectly with a 10k account balance then unexpectedly fail with a 1m balance, because NNs do not always extrapolate well Xu et al, 2020.

You could use "current unrealized gain" / "current account balance" instead, so it will always be in the interval [0,1].

$\endgroup$
9
  • $\begingroup$ Thank you for your response. So, are you saying though, that there is no way to normalize this kind of information? Only clever attempts at keeping it within a reasonable range? $\endgroup$ Jun 10 at 22:07
  • $\begingroup$ Could you clarify precisely what you mean by normalization? It could mean scale to the range [0,1]; or scale and shift to mean 0, variance 1; or other things. $\endgroup$
    – Lee Reeves
    Jun 10 at 23:16
  • $\begingroup$ By normalization, I mean applying any transformation (including ones you mentioned) so that the scale/magnitude of these inputs does not vary hugely from all the other inputs (which, in my case, have been normalized by subtracting the mean and dividing by the standard deviation). $\endgroup$ Jun 11 at 6:13
  • $\begingroup$ I'm wondering if there is any standard method/practice of how to achieve that end with the kinds of variables that are generated directly as a result of actions the agent has taken. Any method/practice that is more standard or rigorous than a heuristic of just generally being clever to try to make sure the numbers don't get too big or small. I hope this clarifies my question. $\endgroup$ Jun 11 at 6:15
  • 2
    $\begingroup$ @LeeReeves I don't agree with the comment that including a cumul. variable like account balance breaks the Markov assumption. If you include it in the state representation, it's right there, so Markov assumption holds. Of course it does bring the practical issue that your state space becomes really big (possibly infinite), which is bad for tabular approaches. And the issue of difficulties with normalisation (as is the topic of the question here), which is bad/challenging/annoying for RL with function approximation. $\endgroup$
    – Dennis Soemers
    Jun 11 at 18:02
2
$\begingroup$

I'll start with the literal question in the title:

Do I need to normalize all state-space variables?

You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary, in practice though. It can be useful just for faster learning, it can be important for numeric stability, or just make it much easier to tune hyperparameters (like a learning rate).

If it's not clear exactly how you could go about doing the normalisation, or if you have multiple different possible solutions, ideally you'd be able to empirically validate what works best for your case by trying them all and running experiments. Maybe this will simply lead to the conclusion that not normalising at all happens to work best for your particular setup.


I am not aware of any 100% robust, standard normalisation or standardisation approaches that work all the time when you have no prior knowledge about upper/lower bounds on the values that your variables might take (or if you do have prior knowledge that there exist no such bounds). I imagine that the online tracking of mean and standard deviation, as suggested by Taw, could work well in practice, especially if your state variables don't explode too wildly at deployment time and suddenly take very different values from what you observed at training time. On the other hand, if such extreme differences do suddenly occur, the approach won't be robust against that. Similarly, you could also track the minimum and maximum observed values, and normalise to a $[0, 1]$ range based on that (but in practice you might exceed that range again if after training you observe more extreme values than you did during training).


I also really like the general idea from Lee Reeves' answer about trying to model your state variables in a different way, such that they still encode the information necessary for your agent to work well, but in a manner that has less of the annoying numeric issues. This requires domain knowledge of your particular problem though.

a simple example would be like if a robot was being trained to pick up balls, and one of the observations was "current number of balls picked up"

In this simple example, why is that observation a relevant observation? My domain knowledge says it probably isn't a relevant observation at all and should be removed. If the goal continues to be to pick up as many balls as possible, why should the robot care about how many balls it already picked up before?

  • If it shouldn't care, just remove the feature instead of trying to normalise it.
  • Maybe it could be relevant if the robot also knows how many balls were lying on the floor at the start; then, knowing how many were already picked up tells you how many are still on the floor. But the same information could be represented as a value in $[0, 1]$ giving you the proportion of balls already picked up... that could work better then!

For example, I have included things like "current account balance" and "current unrealized gain" in my observation space (as I believe it's useful).

For this example, it'd be good to try to think of why you believe such variables would be useful, and then try to again find other ways to encode it (if possible). $\frac{\text{unrealised gains}}{\text{account balance}}$ as suggested by Lee Reeves could be a good way. A series of binary variables that provide your agent with information about the orders of magnitude of these values could be very useful too, for example:

  • $b_1$: is the account balance above $10$?
  • $b_2$: is the account balance above $100$?
  • $b_3$: is the account balance above $100$?
  • $b_4$: is the account balance above $1000$?
  • $b_5$: is the account balance above $10,000$?
  • $b_6$: is the account balance above $100,000$?

I suspect you wouldn't have to keep going too far like that, probably beyond a certain point you stop caring and your agent just knows that it's very rich and that's probably all it needs to know?

$\endgroup$
2
  • 1
    $\begingroup$ Thank you very much for your in-depth response. I will try all of these approaches. It's quite hard choosing an answer to accept here, as every answer adds an additional layer of perspective! After a lot of thought, I'm going to accept Taw's answer as I think it most directly answers the question in the way that I originally envisioned/expected it. $\endgroup$ Jun 15 at 16:06
  • $\begingroup$ I don't know why the so-called AI algorithms require humans to consider such complex problems, this is not artificial intelligence, it is just artificial programming. $\endgroup$
    – Joe Huang
    Jul 15 at 8:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .