I was trying to understand the definition of 2d convolutions vs 3d convolutions. I saw the "simplest definition" according to Pytorch and it seems the following:

  • 2d convolutions map $(N,C_{in},H,W) \rightarrow (N,C_{out},H_{out},W_{out})$
  • 3d convolutions map $(N,C_{in},D,H,W) \rightarrow (N,C_{out},D_{out},H_{out},W_{out})$

Which make sense to me. However, what I find confusing is that I would have expected images to be considered 3D tensors but we apply 2D convolutions to them. Why is that? Why is the channel tensor not part of the "definitionality of the images"?

I also asked this question at https://forums.fast.ai/t/what-is-the-difference-between-2d-vs-3d-convolutions/52495.


Looking at it from the perspective of input to output in that fashion is probably not the best. So lets start with our goal and how these ND convolutions accomplish that (Note these are in my own words, and may not be best stated).

Assumption: There exists highly correlative local associations

Goal: Have a linear model that takes advantage of these local associations

Solution: The ND Convolution

Explanation: ND convolutions take advantage of our locality assumption by connecting only local nodes/neurons. The fact that its a sliding window allows us to learn filters for any location along with ones that can be reusable.

Your Question: Where does the N in ND convolution matter?:
When using an ND convolution we are working off the assumption that there exists this locality in N dimensions and nothing more. So we connect all other components of the input in a dense matter because we have no assumptions to work on in this space. So now going to the shapes you mentioned such as the input and output of the 2D convolution. We are convolving a filter of size $(C_{in}, k_h, k_w)$ with a $(C_{in},H,W)$ activation shaped map (the $N$ just refers to the number of activation maps, and there is no association between them). We use $C_{in}$ channels on the kernel because we are not making any assumptions about locality between channels. On the other hand in a 3D convolution we make locality assumption is 3 dimensions, so our kernel will be ($C_{in}, k_h, k_w, k_r$). These kernel sizes are actually determined by the input size, the amount of dimensions of the kernel will actually match the inputs (minus the batch) because it needs to densely match each one.

You may be thinking now, that in torch this is not the case: This is because its rare to want a 2D Convolution with an inputs that's different than the one you mention, so they only implemented it for a singular shape. I hope this clears up the convolutions and helps you understand not just for 2 and 3 dimensional convolutions, but for all N.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.