Are ANNs just highly distributed lossy compression schemes?

Question

Conceptually speaking, aren't artificial neural networks just highly distributed, lossy compression schemes?

They're certainly efficient at compressing images.

And aren't brains (at least, the neocortex) just compartmentalized, highly distributed, lossy databases?

If so, what salient features in RNNs and CNNs are necessary in any given lossy compression scheme in order to extract the semantic relations that they do? Is it just a matter of having a large number of dimensions/variables?

Could some kind of lossy Bloom filter be re-purposed for the kinds of problems ANNs are applied to?

Answer 1

Auto-encoders, a family of ANNs, are trained with exactly compression in mind. So definitely some ANNs are compressors.

Also, in general, ANNs learn the best concepts to minimize fitness error. I would say that that means, both in classification and regression, to 1) differentiate between various inputs and 2) output the proper value for each input. Point 1), in particular, means having as many distinct net activation configurations as necessary to (at least) tell apart inputs needing (significantly, not in a statistical sense, just a qualitative sense) different outputs. I am inclined to think that if the input is complex enough, you would call what happens "compression".

Answer 2

ANNs don't compress, they generalise.

Often this leads to compression, i.e. the internal generalised representation is smaller than the original input, but not necessarily. Imagine a ANN that is trained to use the screen input of a computer game to play it. If the game is very rich and conceptually deep the internal representation of a single screen input might be a lot bigger than the input itself, because ANNs put the single data points into the context of the overall data. Which leads us to the second point:

ANNs (and the neocortex) model data hierarchically.

This is what makes them so powerful. So it is not just about having a large number of parameters, they also have to be arranged in such a way that they capture the structure of the data (or the world), which very often seems to be hierarchical. Just look a two different pictures of a duck. On the pixel level they might be as different as random images, all the similarities emerge in higher levels of the hierarchy, when enough pixels combined give you the patterns of feathers, beak and webs. Bloom filters obviously lack this property. They would only give you a pixel by pixel account of whether you have seen (almost) exactly this picture before.