I am trying to train networks to achieve what I expected to be a trivial task: learn the identity mapping. However, this is very hard to achieve, and the optimization is hard. Moreover, I don't want to learn $f_\theta(x)=x\;\;\forall x$, but only $f_\theta(x_1)=x_1$ for one particular example (I am trying to overfit!).
Any ideas why this optimization is so hard? The learning curves oscillate and most of the points are not visually satisfactory.
Code is here: https://colab.research.google.com/drive/1umys2gxJ8arodQ0PhLf5TFicfPx0mh74?usp=sharing