Would convolutional NN recognize patterns in encoded images?

Question

I have a set of images that I already trained a CNN to classify successfully. I wonder if it would be possible to encode the images (using XOR in combination with a key of the same length as the image) and train a new net on them.

Thinking logically, the features still exist in the same relation to each other, just in a different form (encoded). Considering that neural networks are incredible at pattern recognition, I assume that it would still be doable.

For people, who cannot imagine how a xor-encoded image would look like:

For a human, it may look like rubbish, but the information is definitely there.

Would love to read your opinion.

Answer 1

First you should give it a try, because anyone's guess could be off, as there isn't really a complete high level analytic model of how neural networks behave on real data. Most results with neural networks are informed by theory, but involve a whole lot of experimental testing.

I suspect your network might at least learn something from the new images, but would struggle to get anything like the same accuracy as without the noise, because the CNN filters rely on being able to detect similar features at different positions. In your scrambled image, there will not be any meaningful and consistent edges/corners etc that a single learned feature detector could learn to match (and therefore present to the next layer).

A fully-connected network would not have this limitation, and would learn just as well on a set of binary features that have been xor'd identically in each position for each example, as it did on the original copy (i.e. only if the picture was 1 bit depth). It would learn less well if each feature was a scaled 8-bit pixel value that was xor'd with the same 8 bit random number in each pixel position, because that would introduce many more non-linear mappings between input and output. Of course a fully-connected network will generally not learn image tasks as well as CNNs in the first place . . . but if it could learn anything useful at all for your image problem, then it will probably out-perform CNNs after the scrambling effect.

As a CNN usually has a few fully-connected layers, then it may be possible to get something from your scrambled images.

Thinking logically, the features still exist in the same relation to each other, just in a different form (encoded)

In terms of being recognisable in the way that a CNN filter extracts them, then the features do not exist. That is a problem.

Answer 2

The answer to your question depends on the nature of the noise that you have xor-ed the image with. If it is the case that the noise is random (or pseudorandom in the formal sense), then it is provably the case that the original pattern will not be learnable in the statistical learning theory sense; this scenario is equivalent to the application of a one-time pad.

To quote the relevant Wikipedia article:

One-time pads are "information-theoretically secure" in that the encrypted message (i.e., the ciphertext) provides no information about the original message to a cryptanalyst (except the maximum possible length[16] of the message). This is a very strong notion of security first developed during WWII by Claude Shannon and proved, mathematically, to be true for the one-time pad by Shannon about the same time.

Answer 3

My gut feel based on the paper I will mention below is that YES, if you apply the same XOR operation on the train and test data, you will be able to train a very "accurate" classifier.

To elaborate on my "gut" feel, please allow me to introduce to you what I personally think is one of the most important paper that came out this year(in fact this paper won the best paper award at ICLR 2017):

Understanding deep learning requires rethinking generalization.

In this paper, the authors showed that deep learning models will generalize to "any" datasets. To give an example of the sort of experiment they conducted on this paper:

• They randomly shuffled the training and test set's labels around in such a manner that for example some images of cats were labeled as dogs whiles some dogs were named cats whilst some cats and dogs images remained correctly labeled. Now it is well understood that deep learning models(including CNNs) are quite resistant to a few noisy labels but in the experiments conducted in the paper mentioned above this was a significant amount of noisy which begs the question why neural networks still performed well on what ended up being a garbage dataset.

The moral of the story is that contrary to what most researchers believed in the past namely that deep learning models magically discover lower level features, middle-level features, and higher-level features hidden within the dataset more like the V1 system of the mammalian brain by learning to compress data, they seem to just memorize anything you give them, including random data.

In short the paper mentioned above showed that deep learning models generalize well to completely random noise(in your case, think images generated from random pixels). Deep learning models will generalize well to anything, anything. And if they can generalize to random data which have no structure, then images that underwent a fixed, predefined transformation like XOR have nothing to a deep learning model.

I must say, this are very worrying findings - to me at least.