I would like to use the bottleneck layer of U-Net (the last layer of the encoder) to calculate the similarity between two images. For that, I have to somehow flatten the last layer of the encoder. In my opinion, there are two approaches:
Take the last layer which in my case is $4 \times 4 \times 16$ and flatten it to 1D
Add a dense before the decoder and then reshape the dense 1D layer into 3D
For the second case, I am not sure how this would affect the network. Arbitrarily reshaping a 1D array into a 3D tensor. Could that introduce weird artifacts? Does someone have experience in a similar problem?