# Would the reward normalization be wrong in early episodes?

It's confusing me that how can we normalize the reward without actually knowing the true mean and variance of the reward distribution, specifically, at the early steps and episodes. This may cause problem for the RL algorithms that use the replay buffer such as DDPG, because this wrongly calculated rewards can stay in buffer for too long and the network will adapt with them. Is there something that I am missing or misunderstood? For algorithms with replay buffer, using standardization is better that normalization?