1
$\begingroup$

I'm trying to replace the strided convolutions of Keras' MobileNet implementation with the ConvBlurPool operation as defined in the Making Convolutional Networks Shift-Invariant Again paper. In the paper, a ConvBlurPool is implemented as follows:

$$ Relu \circ Conv_{k,s} \rightarrow Subsample_s \circ Blur_m \circ Relu \circ Conv_{k,1} $$ where k is the convolution's output kernels, s is the stride, m is the blurring kernel size and the subsample+blur is implemented as a strided convolution with a constant kernel.

My issues start when batch normalization enter the picture. In MobileNet, a conv block is defined as follows (omitting the zero-padding):

$$ Relu \circ BatchNorm \circ Conv_{k,s} $$

I am leaning towards converting it to:

$$ Subsample_s \circ Blur_m \circ Relu \circ BatchNorm \circ Conv_{k,1} $$

i.e., putting the BN before the activation as it's normally done. This is not equivalent though, because the first BN operates on the downsampled signal.

Another possibility would be:

$$ BatchNorm \circ Subsample_s \circ Blur_m \circ Relu \circ Conv_{k,1} $$

with the BN as last operation. This is also not equivalent, because now the BN comes after the ReLu.

Is there any reason to prefer one option over the other? Are there any other options I'm not considering?

$\endgroup$
0
$\begingroup$

After finding the paper authors' Github, I saw that, although they only have a MobileNet V2 model implemented, they choose the Subsample-after-ReLu option (the first one in the question). Although this doesn't fully answer my question, I'll take "the paper authors do it this way" as enough reason to prefer this over the alternative.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.