3 votes
Accepted

Is it a requirement/recommendation to normalize my inputs into [0,1] range?

Generally, between -1 and 1 are ideal, though you can get away with a wider range. For example, using the z-score as the range, you will be outside of this range, sometimes by quite a bit (say, -30 ...
David Hoelzer's user avatar
3 votes

Should I apply normalization to the observations in deep reinforcement learning?

The use of normalisation in neural networks and many other (but not all - decision trees are a notable exception) machine learning methods, is to improve the quality of the parameter space with ...
Neil Slater's user avatar
  • 32.1k
3 votes

Does it make sense to apply batch normalization to a batch size of 1?

It's not possible to use batch normalization with a batch size of 1. Batch normalization requires you to calculate the variance of activation values in the current batch, and variance is undefined for ...
Johann's user avatar
  • 31
3 votes

What if each sample was normalized on its own before sending them to the neural network?

It depends on how your data and prediction task are structured. If you normalize per sample, you lose all relative information between samples. On the other hand, if your data contains a lot of ...
besterma's user avatar
2 votes

Do I need to normalize all state-space variables? If so, how?

I'll start with the literal question in the title: Do I need to normalize all state-space variables? You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary,...
Dennis Soemers's user avatar
  • 10.3k
2 votes
Accepted

Do I need to normalize all state-space variables? If so, how?

The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization ...
Taw's user avatar
  • 1,251
2 votes

Do I need to normalize all state-space variables? If so, how?

It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that. But it might cause problems later, if the values move ...
Lee Reeves's user avatar
2 votes
Accepted

Which generalization of standard deviation to use for multidimensional input normalization

This idea is sometimes applied in computer vision, under the name of Whitening Transform, or ZCA sphering transform. The name whitening comes from signal processing, since removing correlation from a ...
Edoardo Guerriero's user avatar
2 votes

Should I apply normalization to the observations in deep reinforcement learning?

On creating custom environments: ... always normalize your observation space when you can, i.e., when you know the boundaries (From stable-baselines) You could normalize them as part of the ...
mugoh's user avatar
  • 531
2 votes

What does linear regime of nonlinearity mean in normalisation?

The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line. The idea is the same as in electronics: https://en.wikipedia....
Jaume Oliver Lafont's user avatar
2 votes
Accepted

Is it necessary to standardise the expected output

It depends, as mentioned in the comments, on your model and labels. For example, how would you use standardization on a multi-classification problem? Generally, standardization is more favorable for ...
pedrum's user avatar
  • 313
2 votes

Must I "prime" my normalizer with the same data I trained it with in order to use it?

Yes that is indeed the case. You must store the used normalization weights, and use them also with the new data. Otherwise their distribution would be different, and your model's performance will be ...
NikoNyrh's user avatar
  • 767
2 votes
Accepted

What's $\mathbb{V}[\gamma]$ in Return-based Scaling: Yet Another Normalisation Trick for Deep RL?

From the paper, note 3 of section 3.1. Note that, perhaps unconventionally, we treat $\gamma$ as a random variable here, because it is zero at the final step of an episode (even when it is constant ...
Venna Banana's user avatar
2 votes

Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I apply the z-score normalisation directly?

There is, beside of numerical losses not difference between directly using z-normalization and first min-max and then z-normalization. Explanation Both are affine transformations and a combination of ...
Broele's user avatar
  • 561
2 votes
Accepted

What are the consequences when we multiply, instead of add, a penalty term?

Well let's consider one, Ridge regression. We have 2 terms: the regression loss $L^{pred} = \sum(f(x) - y)^2$, which we can see that it is a sum of squared values, thus $L^{pred} \ge 0$ the ...
Alberto's user avatar
  • 1,905
1 vote

Paper to cite about normalizing the inputs to a neural network

I have found a related part in C. C. Aggarwal. Neural Networks and Deep Learning: A Textbook. Springer International Publishing AG, 2018. isbn: 978-3-319-94462-3. doi: 10.1007/978-3-319-94463-0. ...
ha7ilm's user avatar
  • 109
1 vote

Must I "prime" my normalizer with the same data I trained it with in order to use it?

Given the answer provided by @NikoNyrh is correct, I just want to add how you can make a custom pre-processing layer with tensorflow that you can integrate in your model. Assume you want to build a ...
Luca Anzalone's user avatar
1 vote

Can we model the global statistics of the dataset features using LayerNorm?

Layer normalization normalizes a batch of data $X$ (usually the output of some intermediate hidden layer) with dimensionality $D$ (e.g. $D$ features/columns), independently on each dimension ...
Luca Anzalone's user avatar
1 vote

In the attention mechanism, why don't we normalize after multiplying values?

Because what attention does is to control how much of the information in $V$ to use based on weights computed through the similarity between $Q$ and $K$. When we multiply the attention weights by $V$, ...
Areg Sarvazyan's user avatar
1 vote

During batch normalization is the mini-batch gone through twice, one to calculate the mean and variance and then to normalize them?

I think it'd be helpful to refer to the batchnorm formula given in the PyTorch implementation. In particular, given an input $x$, you would get the mean and variance ($\mathbf{E}[x]$ and $\text{Var}[x]...
PeaBrane's user avatar
  • 356
1 vote

Normalization of possibly not fully representative data

A friend of mine answered on this question in different social media on different language, I'll post his answer here: 1. scaler should be saved in this case. You do fit_transform in the example the ...
banderlog013's user avatar
1 vote

How to normalize images before training?

tl;dr subtracting the mean and dividing by the standard deviation is theoretically more sound, but is impractical compared to dividing by $255$. Explanation As you know neural networks perform better ...
Djib2011's user avatar
  • 3,183
1 vote

Is data leakage relevant when scaling across samples?

Considering that you are making a minmax scaling, the only time in which there would be no risk of data leakage is if the minimum value on your training set equals the minimum value of the test set, ...
Moonstone5629's user avatar
1 vote
Accepted

Flatten image using Neural network and matrix transpose

Yes, if you have 3 images (and by images I assume you mean samples) the flatten layer will be of the shape $12288*3$ ($64*64*3=12288$). The size of $W$ however does not change, and nor does the size ...
Recessive's user avatar
  • 1,396
1 vote

What would be a typical pre-processing and data normalization pipeline for time series data (for non-linear models such as neural networks)?

I would say any normalization such as min-max or standard deviation is fine as far as the scaling factor is provided as a feature, since time-series of different scale might behave differently.
F4RZ4D's user avatar
  • 86

Only top scored, non community-wiki answers of a minimum length are eligible