Consider the following game on a MNIST dataset:
- There are 60000 images.
- You can pick any 1000 images and train your Neural Network without access to the rest of images.
- Your final result is prediction accuracy on all dataset.
How to formalize this process in terms of information theory? I know that information theory works with distributions, but maybe you can provide some hints how to think in terms of datasets instead of distributions.
- What is the information size of all datasets. My first idea was that each image is iid from uniform distribution and information content = -log2(1/60000). But common sense and empirical results (training neural network) show that there are similar images and very different images holding a lot more information. For example if you train NN only on good looking images of 1 you will get bad results on unusual 1s.
- How to formalize that the right strategy is to choose as much as possible different 1000 images. I was thinking to take image by image with the highest entropy relative to the images you already have. How to define distance function.
- How to show that all dataset contains N bits of information, training dataset contain M bits of information and there is a way to choose K images < 60000 that hold >99.9% of information.