Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
When training a neural network, we often run into the issue of overfitting.
However, is it possible to put overfitting to use? Basically, my idea is, instead of storing a large dataset in a database, you can just train a neural network on the entire dataset until it overfits as much as possible, then retrieve data "stored" in the neural network like it's a hashing function.
The auto-encoder (AE) can be used to learn a compressed representation (a vectorised hash value) of each observation in the training dataset, $z$, which can then be used to later retrieve the original (or similar) observation.
The variational auto-encoder (VAE), a statistical variation of AE, can also be used to generate objects similar to the observations (or inputs) in the training set.
There are other data compressor models, for example, Helmholtz machine, which precedes both the AE and VAE.
Train a network that has large input and small output. Turn it upside down (yes, you may do that). By giving the small outputs, corresponging to input, the ideally- trained network will generate those large data. But you see in all compression there will be data lost, so generated data will be slightly :DD different then original dataset. So its suitable for statistical data, like images, whatever, but not for structured like text or the most unsuitable example - program source code.
An Auto-Encoder is probably what you are looking for. AE is a very powerful Neural Network when you want to compress data and get a lower dimensional representation of the data with maximum information retained.
If you are interested to know how it works -- Think of it as training a NN to predict the data itself. Which means your input and output layers would exactly be the same. So how does this help in compression ? The Hidden Layer -- Here is where the magic happens. The compression comes from the fact that we use lesser number of neurons in the hidden layer than in the i/p layer. Assuming that the NN we build nicely predicts the i/p at the end of the training, the output of the hidden layers can be thought of as newly engineered features which are less in dimension yet powerful enough to represent the information in original data.
See What is the difference between encoders and auto-encoders? For a working example of a neural network being used to compress (and decompress) images.
