4
$\begingroup$

I was reading the "Deep Learning with Python" by François Chollet. He mentioned separable convolution as following

This is equivalent to separating the learning of spatial features and the learning of channel-wise features, which makes a lot of sense if you assume that spatial locations in the input are highly correlated, but different channels are fairly independent.

But I could not understand what he meant by saying "correlated spatial locations". Can some explain what he means or the purpose of separable convolutions? (except performance-related part).

Edit: Separable convolution means that first depthwise convolution is applied then pointwise convolution is applied.

Improve this question
$\endgroup$
3
  • $\begingroup$ To explain this part "spatial locations in the input are highly correlated", i.e. that pixels in some neighbourhood of a certain channel are likely to be correlated, imagine that in some neighbourhood of the channel you have the face of a person, then it should be clear that all pixels that belong to this face are correlated, in the sense that they will have very similar intensity value (for the respective channel). I can't answer why channels would be independent without more context. Maybe they would be independent because they are not encoding the RGB values but something else. $\endgroup$
    – nbro
    Oct 16, 2020 at 22:56
  • 1
    $\begingroup$ @nbro It does not have to be RGB, lets say we apply separable convolution as second layer. In this context channel number comes from number of filter we applied as you probably know? The example you gave did not clarify the problem. $\endgroup$
    – Enes
    Oct 17, 2020 at 13:47
  • $\begingroup$ Why doesn't my example clarify what the authors mean by "spatial correlation"? $\endgroup$
    – nbro
    Oct 17, 2020 at 15:25

1 Answer 1

Reset to default
3
$\begingroup$

Context of the question

This is a link to the text cited in the question.

It refers to the usage of SeparableConv2D (tf, keras name). A related question on StackOverflow is "What is the difference between SeparableConv2D and Conv2D layers". This answer points to this excellent article by Chi-Feng Wang:

A Basic Introduction to Separable Convolutions

Answer to the question

In image processing, a separable convolution converts a NxM convolution to two convolutions with kernels Nx1 and 1xM. Using this idea, in NN a SeparableConv2D converts a WxHxD convolution (width x height x depth, where depth means number of incoming features ) to two convolutions with kernels WxHx1 and 1x1xD.

Note the first kernel doesn't handles information across features, thus, it is "learning of spatial features". The 1x1xD kernel doesn't handles different points, it is "learning of channel-wise features".

About the phrase "spatial locations in the input are highly correlated", my understanding of what the author means is: Assume we have a channel (feature) image that each pixel measures the "distance to the background". When we pass from one pixel to a neighbors one, it is expected some continuity in the value (except for edge pixels): correlation. Instead, if we have a channel that measures "brightness" and another one that measures "distance to background" the two values for one specific pixel has little correlation.

Finally, about title question "When should we use separable convolution?" : if the final output must depend of some features of one pixel and some other features of neighbors pixels in a very unpredictable way, a complete WxHxD convolution must be used. However if, as is more usual, you can handle first spatial dependencies (neighborhood) to extract pixel features and next handle pixel-by-pixel these features to get the output, better use a WxHx1 followed by 1x1xD, saving lots of network parameters, thus, saving training time.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .