9
$\begingroup$

I am trying to understand what channels mean in convolutional neural networks. When working with grayscale and colored images, I understand that the number of channels is set to 1 and 3 (in the first conv layer), respectively, where 3 corresponds to red, green, and blue.

Say you have a colored image that is $200 \times 200$ pixels. The standard is such that the input matrix is a $200 \times 200$ matrix with 3 channels. The first convolutional layer would have a filter that is size $N \times M \times 3$, where $N,M < 200$ (I think they're usually set to 3 or 5).

Would it be possible to structure the input data differently, such that the number of channels now becomes the width or height of the image? i.e., the number of channels would be 200, the input matrix would then be $200 \times 3$ or $3 \times 200$. What would be the advantage/disadvantage of this formulation versus the standard (# of channels = 3)? Obviously, this would limit your filter's spatial size, but dramatically increase it in the depth direction.

I am really posing this question because I don't quite understand the concept of channels in CNNs.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

Say you have a colored image that is 200x200 pixels. The standard is such that the input matrix is a 200x200 matrix with 3 channels. The first convolutional layer would have a filter that is size $N×M×3$, where $N,M<200$ (I think they're usually set to 3 or 5).

Would it be possible to structure the input data differently, such that the number of channels now becomes the width or height of the image? i.e., the number of channels would be 200, the input matrix would then be 200x3 or 3x200. What would be the advantage/disadvantage of this formulation versus the standard (# of channels = 3)? Obviously, this would limit your filter's spatial size, but dramatically increase it in the depth direction.

The different dimensions (width, height, and number of channels) do have different meanings, the intuition behind them is different, and this is important. You could rearrange your data, but if you then plug it into an implementation of CNNs that expects data in the original format, it would likely perform poorly.

The important observation to make is that the intuition behind CNNs is that they encode the "prior" assumption or knowledge, the "heuristic", the "rule of thumb", of location invariance. The intuition is that, when looking at images, we often want our Neural Network to be able to consistently (in the same way) detect features (maybe low-level features such as edges, corners, or maybe high-level features such as complete faces) regardless of where they are. It should not matter whether a face is located in the top-left corner or the bottom-right corner of an image, detecting that it is there should still be performed in the same way (i.e. likely requires exactly the same combination of learned weights in our network). That is what we mean with location invariance.

That intution of location invariance is implemented by using "filters" or "feature detectors" that we "slide" along the entire image. These are the things you mentioned having dimensionality $N \times M \times 3$. The intuition of location invariance is implemented by taking the exact same filter, and re-applying it in different locations of the image.

If you change the order in which you present your data, you will break this property of location invariance. Instead, you will replace it with a rather strange property of... for example, "width-colour" invariance. You might get a filter that can detect the same type of feature regardless its $x$-coordinate in an image, and regarldess of the colour in which it was drawn, but the $y$-coordinate will suddenly become relevant; your filter may be able to detect edges of any colour in the bottom of an image, but fail to recognize the same edges in the top-side of an image. This is not an intuition that I would expect to work successfully in most image recognition tasks.

Note that there may also be advantages in terms of computation time in having the data ordered in a certain way, depending on what calculations you're going to perform using that data afterwards (typically lots of matrix multiplications). It is best to have the data stored in RAM in such a way that the inner-most loops of algorithms using the data (matrix multiplication) access the data sequentially, in the same order that it is stored in. This is the most efficient way in which to access data from RAM, and will result in the fastest computations. You can generally safely expect that implementations in large frameworks like Tensorflow and PyTorch will already require you to supply data in whatever format is the most efficient by default.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Thanks for the detailed answer. To clarify, when I said to "rearrange the data," I mean that the CNN is ALSO trained under the new format. I am not suggesting that we use a CNN trained on the previous format and use it to make a predictions on an image in the new format. Apologies for the miscommunication. As a side note, this question was inspired by my question in datascience.stackexchange.com/questions/43291/… where I am trying to use CNNs on non-image data. $\endgroup$
    – anonuser01
    Dec 30, 2018 at 19:07
  • 2
    $\begingroup$ @Iamanon Yes that's what I assumed in my answer as well, that you'd do both training and evaluation with the "rearranged" data. I still expect that would generally perform less well. The dimensions that your filters are "sliding" along should by the dimensions where you want the "invariance". That should be in the spatial dimensions (width and height), but NOT in other dimensions (e.g. channels for different colours). $\endgroup$
    – Dennis Soemers ♦
    Dec 30, 2018 at 19:46
  • 2
    $\begingroup$ @Iamanon So, intuitively you could say that dimensions for which you expect the invariance property to be beneficial are not "channels", and all other dimensions are "channels". For example, TensorFlow has conv2d and conv3d for inputs with invariance in 2D or 3D spaces (and any other input dimensions are "channels"). It also has convolution (tensorflow.org/api_docs/python/tf/nn/convolution) with an arbitrary number input_spatial_shape of "spatial" inputs, and an arbitrary number in_channels of remaining input dimensions. $\endgroup$
    – Dennis Soemers ♦
    Dec 30, 2018 at 19:55
  • 2
    $\begingroup$ @Iamanon For your physics application... it's difficult for me to judge where you'd want the property of invariance. I could easily see it being a useful property in all of the dimensions. Spatial and temporal invariance both can make sense. You have to ask yourself: whatever kind of high-level features I expect my NN to "learn inside hidden layers", should the manner in which it detects them be invariant to time and/or space? If yes, filters should slide along those dimensions. If no, treat them as "channels". $\endgroup$
    – Dennis Soemers ♦
    Dec 30, 2018 at 20:01
  • 1
    $\begingroup$ Thanks again. The idea that the channels are NOT the dimension(s) in which you expect the invariance property makes sense! What happens if there is more than one dimension that fits under this idea of channels? It seems these deep learning libraries only allow for a single input as the "channels." With one dimension, say, of size 5, you would simply set channels = 5. What if you had some data where you had 2 dimensions of size 5 and 10, respectively, in which you are NOT expecting the invariance property, how would you define the number of channels? $\endgroup$
    – anonuser01
    Dec 30, 2018 at 20:37
1
$\begingroup$

If you have a gray scale image, that means you are getting data from one sensor. If you have an RGB image, that means you are getting data from three sensors. If you have a CMYK image, that means you are getting data from four sensors.

So, channels can be considered as same information seen from different perspective. (here color)

If you see how the kernel (for example 5*5*3) moves, it moves only in XY direction and not in the channel direction. So, you are trying to learn features in XY direction from all the channels together.

But, if you exchange the dimensions like you mentioned your XY dimensions become 200*3 or 3*200 and your channels become 200. In this case you are moving kernel not in the actual XY spatial space of image. So, it doesn't make any sense according to me. You are contradicting the basic concept of CNN by doing so.

The concept of CNN itself is that you want to learn features from the spatial domain of the image which is XY dimension. So, you cannot change dimensions like you mentioned.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .