1
$\begingroup$

I have a structured dataset of around 100 gigs, and I am using DNN for classification in TF 2.0. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in batches to train the model.

Now, the input to the network should be normalized and for that, I need training dataset mean and SD. I have been reading TensorFlow docs to get info on how to normalize features when reading data in batches. But, couldn't find one. though I found this article, it is only for the case where entire data can be loaded in memory.

So, If any of you have worked on creating such a TensorFlow data pipeline for normalizing input features while loading data in batches and training model, It would be helpful.

$\endgroup$
0
$\begingroup$

One way could be to first iterate over the dataset in batches, just to get the mean and sd. Then when running training, use the true population parameters obtained before.

$\endgroup$
0
$\begingroup$

The central limit theorem tells us that the error in an estimate of the mean or standard deviation of a dataset will decline as $\frac{1}{\sqrt{n}}$, where $n$ is the number of samples taken at random from the set, and combined together to compute the mean and deviation.

If you select at random, for example, $10^6$ examples (probably a few megabytes), then the mean and standard deviation you compute from those examples will be within $\frac{1}{sqrt(10^6)} = \frac{1}{10^3}$ of the "true" answer. That's one part in 1,000, which is certainly accurate enough to use for re-scaling the dataset.

$\endgroup$
  • $\begingroup$ I was thinking along the same lines, To use certain number of observation for getting a ball park number, and now I could also quantity the error! But isn't there any more elegant solution to this? $\endgroup$ – thisisbhavin Nov 4 at 16:44
  • $\begingroup$ You can also calculate mean and standard deviation using single-pass "streaming" approaches. Books like numerical.recipes cover these algorithms, and many more statistical algorithms for single-passes through batches of a dataset. However, the central limit theorem tells us these will probably not improve your accuracy enough to be worthwhile. Why spend many minutes (or perhaps hours) reading through all the data for a 0.1% improvement in a statistic you use only for normalization (which doesn't need to be that accurate)? $\endgroup$ – John Doucette Nov 4 at 16:53
  • $\begingroup$ Indeed, its better to invest that time doing something else that could actually help improve accuracy. Thank you $\endgroup$ – thisisbhavin Nov 4 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.