1
$\begingroup$

I have been trying to train a CNN for the super-resolution task based on the work of Dong et al., 2015 [1]. The network structure built in PyTorch is as follows:

  (0): Conv2d(1, 64, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))
  (1): ReLU()
  (2): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
  (3): ReLU()
  (4): Conv2d(32, 1, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))

I have a training dataset which consists of approximately 22.000 sub-images generated from 91 images and training is performed only on the Y channel of the images in YCbCr color space. During the training process, I used RMSE loss and calculated the PSNR (Peak Signal to Noise Ratio) from that loss. I observed that PSNR value is increasing as a result of decreasing loss as expected and as depicted in the figure.

psnr_vs_epoch_and_loss_vs_epoch

I trained the network for 25 epochs. After 10th epoch, the network is converged and PSNR value started to increase slowly. After this point, I was expecting to get even better visual outputs with higher PSNR values achieved. However, when I analyze the results of the network, there are some black pixels appearing in white spots in the output images that the network produced.

10th_epoch_vs_25th_epoch

After 25-epoch training was completed, I compared the outcome of 25th epoch (right) with that of 10th epoch (left) as you can see in the figure above.

What might be the possible reasons for the undesired black pixels and the possible precautions that can be embedded into the network to get rid of these?

If you would like to check my code, you can visit here.

[1] Dong, Chao, Chen Change Loy, Kaiming He, and Xiaoou Tang. "Image Super-Resolution Using Deep Convolutional Networks." IEEE Transactions on Pattern Analysis and Machine Intelligence 38, no. 2 (2015): 295-307. doi:10.1109/tpami.2015.2439281.

$\endgroup$
3
  • $\begingroup$ I met the similar problem like you proposed. Do you solve it later? Look forward to your help. Thanks. $\endgroup$
    – 吴沛阳
    Mar 24, 2022 at 15:56
  • $\begingroup$ @吴沛阳 unfortunately, still don't know how to solve it. $\endgroup$
    – Utku
    Mar 25, 2022 at 11:44
  • $\begingroup$ How about white dot? $\endgroup$
    – Cloud Cho
    Sep 29, 2023 at 16:14

2 Answers 2

2
$\begingroup$

This is a while ago, but still this problem might occur to someone. I encountered the same problem and found that the reason was how the resulting tensors are transformed to images. It seams that pixels with value 1.0 are shown as white while pixels with values bigger than 1.0 are interpreted as being black sometimes. So adding (in PyTorch)

indices = tensor > 1.0
tensor[indices] = 1.0 

before converting to the image in the visualization method fixed the problem for me.

$\endgroup$
4
  • $\begingroup$ Do you remember which layer in neural network? Was it some tensor in the last layer? Also which Super Resolution method (list at github.com/LoSealL/…)? $\endgroup$
    – Cloud Cho
    Sep 29, 2023 at 16:20
  • $\begingroup$ It was not in a layer at all in that sense. The Network outputs values that are mostly between 1 and 0. But sometimes these values are slightly over 1 or under 0 if no activation function is used after the last layer (e. g. no sigmoid) . So the network gives a value whiter than white so to say. Independent of the super resolution method this might always happen without activation function. However, some visualization packets show values bigger than 1 as black. So you might fix that by clipping the values between 0 and 1 before visualization. Hope that clarifies things. $\endgroup$ Oct 13, 2023 at 17:39
  • $\begingroup$ "But sometimes these values are slightly over 1 or under 0 (at the last layer without activation function)"...doesn't it apply to specific neural network model? I thought the last layer of most CNN implement models is a dense layer. Do you mean the layer before activation in the middle? $\endgroup$
    – Cloud Cho
    Oct 13, 2023 at 18:00
  • $\begingroup$ No, I mean the last layer, which can also be a Convolutional Layer, depending on the architecture. But also a dense layer can output values that are not in the range between 0 and 1. $\endgroup$ Oct 21, 2023 at 14:43
1
$\begingroup$

Great that a solution was found (clamp larger-than-one pixels' brightness before showing the image). But I suggest that you either add a sigmoid activation, or clamp the network's output directly from zero to one. This way the plotting code doesn't need to handle out-of-spec values, and in theory the model can achieve a bit smaller loss. It doesn't need try to predict exactly 1.0 or 0.0, but values like 2.3 and -0.5 are ok since they get clipped to the correct values (assuming that clipping, not sigmoid is used).

Sigmoid has a bit nicer gradients than hard clipping, if the model overshoots predictions and the target value is for example 0.9 or 0.1.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .