Learning the identity function is not trivial at all.

The main reason is that the identity function is a linear, and a neural network try to approximate it in a non linear fashion. Non linear activations in particular compress and expand values that linearly would have the same distance, so they are not suitable to approximate something like the identity function. I see you used a linear activation but the network still learn in a non linear fashion.

Residual Neural Networks (ResNet) were suggested precisely to help a neural network learning the identity function. The skip connection simply does the job of making the output equal to the input, forcing the network to focus in the residuals (output - identity), so for a one layer ResNet learning the identity function become trivial cause the network simply has to learn to push all weights to 0.

But without this trick, approximating the identity is simply very hard, this is why your UNet experiment was not as successful, even with a single training image.

Learning the identity function is not trivial at all.

The main reason is that the identity function is linear, and a neural network try to approximate it in a non linear fashion. Non linear activations in particular compress and expand values that linearly would have the same distance, so they are not suitable to approximate something like the identity function. I see you used a linear activation but the network still learn in a non linear fashion.

Residual Neural Networks (ResNet) were suggested precisely to help a neural network learning the identity function. The skip connection simply does the job of making the output equal to the input, forcing the network to focus on residuals (output - identity), so for a one layer ResNet learning the identity function become trivial cause the network simply has to learn to push all weights to 0.

But without this trick, approximating the identity is simply very hard, this is why your UNet experiment was not successful, even with a single training image.