1
$\begingroup$

I'm training an autoencoder, that does not downsample images but processes them in the same size. For example, a 256x256 input will always be processed at 256x256 resolution, only the channels increase deeper in the network. This design is due to the next stage for which the model will be used. L2 regularization on activations is added since this is essentially a sparse autoencoder.

However, I get some strange artifacts in the corners when the model converges:

Reference Prediction

The left image is the input, the image on the right is the output. The artifacts can be seen on the top-left and top-right corners of the output image.

Can someone explain what causes these artifacts, and how they can be fixed?

$\endgroup$

2 Answers 2

2
$\begingroup$

Assuming that you're using convolutional layers, those artifacts may be related to the boundary conditions used. The convolution kernels have a spatial support of say 3x3 pixels, meaning that the response at a position is a function of the corresponding inputs in a 3x3 neighborhood at that position. If the position is adjacent to a boundary, say at the upper left pixel, then the south-west, west, north-west, north, and north-east pixels are outside the image region.

If you only use so-called "valid" pixels, that is, you only get responses for those positions where the entire convolution kernel is inside the image region, then you're ignoring some information at the "invalid" pixels. On the other hand, we are missing information at those invalid pixels, so an assumption must be made about those missing pixels.

For your application, in the internal layers, where the expected features are gradient-like / oscillating and localized, you may wish to try zero-padding, because that's like an assumption that one doesn't assume new feature information outside the region.

However, at the first layer itself, the convolution is with a natural image (as in your example), so you need to predict the pixel values at the border, so there you might want to experiment with "repeat" or "reflecting"/"mirror" boundary conditions.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you for your response. I'm using "same" padding in all layers, with zero padding. I'll try out the other padding modes in the initial layer, as suggested. $\endgroup$ Mar 22 at 14:56
  • 1
    $\begingroup$ Using "reflect" padding on the image worked great! Thanks! $\endgroup$ Mar 23 at 12:33
3
$\begingroup$

That's a classic checkerboard artifact. I would guess you're using a CNN as encoder/decoder architectures, since is well known that convolution layers, especially at the upsampling phase, cause these kind of issues.

The use of L2 regularization is also a potential culprit. Pixelwise losses are also known to lead to noisy images, since they don't allow reconstruction freedom to the model.

To understand why you can check this blog post, in this case gifs are more understandable than a thousand words, but long story short, is mostly related to emphasized overlapping region generated by the combinations of kernel/stride sizes.

Solutions:

  • use some more complex but not pixelwise regularization like perceptual loss
  • remove ConvTranspose2D layers if you're using any, and move to other upsampling layers like Upsample (with bilinear or cubic interpolation)
  • in my experience a nice trick that works pretty well in production when the artifact are only on the edges or corners is to pad the input image (with reflect setting) and then crop back the generated image in the original size. This way the model generate artifacts only in non important areas of the image.
$\endgroup$
4
  • 1
    $\begingroup$ I'm already using perceptual loss alongside MSE loss. Furthermore, since there is no downsampling done, there is no need for upsampling. I'll try out the input padding trick, thanks. $\endgroup$ Mar 22 at 15:00
  • $\begingroup$ then I would simply remove the MSE loss, or replace it with L1 loss if the perceptual does not suffice. And if you're using an autoencoder I have a hard time believing there's no upsampling happening, how do you go from the latent bottleneck vector size to the original one otherwise? $\endgroup$ Mar 22 at 15:04
  • 1
    $\begingroup$ Please read the question to get the exact setup details. I'll try using L1 loss instead of L2 as suggested. $\endgroup$ Mar 22 at 15:12
  • 1
    $\begingroup$ ok, if you can provide more details about the architecture maybe we can also give more detailed suggestions on that. $\endgroup$ Mar 22 at 15:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .