2
$\begingroup$

As everyone experienced in deep learning might know, in an image classification problem we normally add borders to images then resize it to the input size of a CNN network. The reason of doing this is to keep aspect ratio of the original image and retain it's information.

I have seen people fill black (0 pixel value for each channel), gray (127 pixel value for each channel), or random value generated from gaussian distribution to the border.

My question is, is there any proof that which of these is correct?

$\endgroup$
1
$\begingroup$

If the computational components of the forward feed through the network have no curvature, which is normally the case in a sum of products, then it can be proven that any constant pixel value is equivalent in terms of effect on convergence results. We wouldn't expect a proof for that, since it would be too trivial to spend time writing up for publication. In general, functioning vision systems have feed forward computational components with curvature, so the padding is likely significant.

Even the convolutional layers may have activation functions or something even more complex going forward, as noted in Gauge Equivariant Convolutional Networks and the Icosahedral CNN (Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, Max Welling, 2019).

If purely stochastic values with value distributions like that of the un-padded coordinates are used, it may be possible to prove that some gain is made, but none appeared in a few academic article searches just made. Not surprisingly, there are many proofs regarding the properties of various message padding strategies for cryptography.

Short of the inclusion of thermal or quantum noise acquisition devices in VLSI circuitry and exposure of those devices in software, purely stochastic values cannot be generated. This leaves the risk of a learning approach expected to extract features from frames learning features of the pseudo-random noise generator used to pad.

The answer is that none are universally correct and there appears to be much work to do in proving advantages between different techniques in as many cases as such advantages can be proven.

$\endgroup$
0
$\begingroup$

I've more often seen image resizing than padding to be honest and tend to resize the images. Maybe it's because datasets I've used have images with near equal aspect ratios.

One major exception was when I worked with MR images. These were orthogonal and it would be wrong to mess up the aspect ratio. However, in this domain images have black borders everywhere, so a zero-padding was easy to apply.

The most common use of padding I've seen is for data augmentations (to fill values gone due to translations, rotations, shifting etc.). In this regard, I've used many types of paddings (constant value, random value, 'same' padding, mirrored padding etc.) The best I've found to empirically work is zero-padding but I don't think that you'll ever find a proof for this. I like to think of it as a hyperparameter; different padding strategies maybe work better for different tasks. Though I think that zero-padding is the safest (there is a small chance of messing things up).

$\endgroup$
  • 1
    $\begingroup$ I agree with the point that this question depends on the nature of dataset. I'm working on faces at the moment, so losing the aspect ratio by simply resize would potentially disturb the learning process. I could even do shearing instead. However, there seems to be many ways and I couldn't find any experimental results comparing different strategies. Anyway, thanks for the reply! $\endgroup$ – Eugene Liu Aug 7 at 10:24
  • $\begingroup$ No problem. Good luck :) $\endgroup$ – Djib2011 Aug 7 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.