Learning an identity function with convolutional networks

Question

I am trying to train networks to achieve what I expected to be a trivial task: learn the identity mapping. However, this is very hard to achieve, and the optimization is hard. Moreover, I don't want to learn $f_\theta(x)=x\;\;\forall x$, but only $f_\theta(x_1)=x_1$ for one particular example (I am trying to overfit!).

Any ideas why this optimization is so hard? The learning curves oscillate and most of the points are not visually satisfactory.

Code is here: https://colab.research.google.com/drive/1umys2gxJ8arodQ0PhLf5TFicfPx0mh74?usp=sharing

Answer 1

Learning the identity function is not trivial at all.

The main reason is that the identity function is linear, and a neural network try to approximate it in a non linear fashion. Non linear activations in particular compress and expand values that linearly would have the same distance, so they are not suitable to approximate something like the identity function. I see you used a linear activation but the network still learn in a non linear fashion.

Residual Neural Networks (ResNet) were suggested precisely to help a neural network learning the identity function. The skip connection simply does the job of making the output equal to the input, forcing the network to focus on residuals (output - identity), so for a one layer ResNet learning the identity function become trivial cause the network simply has to learn to push all weights to 0.

But without this trick, approximating the identity is simply very hard, this is why your UNet experiment was not successful, even with a single training image.