# Super Resolution CNN generates black dots on output images

I have been trying to train a CNN for the super-resolution task based on the work of Dong et al., 2015 [1]. The network structure built in PyTorch is as follows:

  (0): Conv2d(1, 64, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))
(1): ReLU()
(2): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
(3): ReLU()
(4): Conv2d(32, 1, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))


I have a training dataset which consists of approximately 22.000 sub-images generated from 91 images and training is performed only on the Y channel of the images in YCbCr color space. During the training process, I used RMSE loss and calculated the PSNR (Peak Signal to Noise Ratio) from that loss. I observed that PSNR value is increasing as a result of decreasing loss as expected and as depicted in the figure.

I trained the network for 25 epochs. After 10th epoch, the network is converged and PSNR value started to increase slowly. After this point, I was expecting to get even better visual outputs with higher PSNR values achieved. However, when I analyze the results of the network, there are some black pixels appearing in white spots in the output images that the network produced.

After 25-epoch training was completed, I compared the outcome of 25th epoch (right) with that of 10th epoch (left) as you can see in the figure above.

What might be the possible reasons for the undesired black pixels and the possible precautions that can be embedded into the network to get rid of these?

If you would like to check my code, you can visit here.

[1] Dong, Chao, Chen Change Loy, Kaiming He, and Xiaoou Tang. "Image Super-Resolution Using Deep Convolutional Networks." IEEE Transactions on Pattern Analysis and Machine Intelligence 38, no. 2 (2015): 295-307. doi:10.1109/tpami.2015.2439281.