# What is the concept of channels in CNNs?

I am trying to understand what channels mean in convolutional neural networks. When working with grayscale and colored images, I understand that the number of channels is set to 1 and 3 (in the first conv layer), respectively, where 3 corresponds to red, green, and blue.

Say you have a colored image that is 200x200 pixels. The standard is such that the input matrix is a 200x200 matrix with 3 channels. The first convolutional layer would have a filter that is size $$N \times M \times 3$$, where $$N,M<200$$ (I think they're usually set to 3 or 5).

Would it be possible to structure the input data differently, such that the number of channels now becomes the width or height of the image? i.e., the number of channels would be 200, the input matrix would then be 200x3 or 3x200. What would be the advantage/disadvantage of this formulation versus the standard (# of channels = 3)? Obviously, this would limit your filter's spatial size, but dramatically increase it in the depth direction.

I am really posing this question because I don't quite understand the concept of channels in CNNs.

Say you have a colored image that is 200x200 pixels. The standard is such that the input matrix is a 200x200 matrix with 3 channels. The first convolutional layer would have a filter that is size $$N×M×3$$, where $$N,M<200$$ (I think they're usually set to 3 or 5).

Would it be possible to structure the input data differently, such that the number of channels now becomes the width or height of the image? i.e., the number of channels would be 200, the input matrix would then be 200x3 or 3x200. What would be the advantage/disadvantage of this formulation versus the standard (# of channels = 3)? Obviously, this would limit your filter's spatial size, but dramatically increase it in the depth direction.

The different dimensions (width, height, and number of channels) do have different meanings, the intuition behind them is different, and this is important. You could rearrange your data, but if you then plug it into an implementation of CNNs that expects data in the original format, it would likely perform poorly.

The important observation to make is that the intuition behind CNNs is that they encode the "prior" assumption or knowledge, the "heuristic", the "rule of thumb", of location invariance. The intuition is that, when looking at images, we often want our Neural Network to be able to consistently (in the same way) detect features (maybe low-level features such as edges, corners, or maybe high-level features such as complete faces) regardless of where they are. It should not matter whether a face is located in the top-left corner or the bottom-right corner of an image, detecting that it is there should still be performed in the same way (i.e. likely requires exactly the same combination of learned weights in our network). That is what we mean with location invariance.

That intution of location invariance is implemented by using "filters" or "feature detectors" that we "slide" along the entire image. These are the things you mentioned having dimensionality $$N \times M \times 3$$. The intuition of location invariance is implemented by taking the exact same filter, and re-applying it in different locations of the image.

If you change the order in which you present your data, you will break this property of location invariance. Instead, you will replace it with a rather strange property of... for example, "width-colour" invariance. You might get a filter that can detect the same type of feature regardless its $$x$$-coordinate in an image, and regarldess of the colour in which it was drawn, but the $$y$$-coordinate will suddenly become relevant; your filter may be able to detect edges of any colour in the bottom of an image, but fail to recognize the same edges in the top-side of an image. This is not an intuition that I would expect to work successfully in most image recognition tasks.

Note that there may also be advantages in terms of computation time in having the data ordered in a certain way, depending on what calculations you're going to perform using that data afterwards (typically lots of matrix multiplications). It is best to have the data stored in RAM in such a way that the inner-most loops of algorithms using the data (matrix multiplication) access the data sequentially, in the same order that it is stored in. This is the most efficient way in which to access data from RAM, and will result in the fastest computations. You can generally safely expect that implementations in large frameworks like Tensorflow and PyTorch will already require you to supply data in whatever format is the most efficient by default.

• Thanks for the detailed answer. To clarify, when I said to "rearrange the data," I mean that the CNN is ALSO trained under the new format. I am not suggesting that we use a CNN trained on the previous format and use it to make a predictions on an image in the new format. Apologies for the miscommunication. As a side note, this question was inspired by my question in datascience.stackexchange.com/questions/43291/… where I am trying to use CNNs on non-image data. – Iamanon Dec 30 '18 at 19:07
• @Iamanon Yes that's what I assumed in my answer as well, that you'd do both training and evaluation with the "rearranged" data. I still expect that would generally perform less well. The dimensions that your filters are "sliding" along should by the dimensions where you want the "invariance". That should be in the spatial dimensions (width and height), but NOT in other dimensions (e.g. channels for different colours). – Dennis Soemers Dec 30 '18 at 19:46
• @Iamanon So, intuitively you could say that dimensions for which you expect the invariance property to be beneficial are not "channels", and all other dimensions are "channels". For example, TensorFlow has conv2d and conv3d for inputs with invariance in 2D or 3D spaces (and any other input dimensions are "channels"). It also has convolution (tensorflow.org/api_docs/python/tf/nn/convolution) with an arbitrary number input_spatial_shape of "spatial" inputs, and an arbitrary number in_channels of remaining input dimensions. – Dennis Soemers Dec 30 '18 at 19:55
• @Iamanon For your physics application... it's difficult for me to judge where you'd want the property of invariance. I could easily see it being a useful property in all of the dimensions. Spatial and temporal invariance both can make sense. You have to ask yourself: whatever kind of high-level features I expect my NN to "learn inside hidden layers", should the manner in which it detects them be invariant to time and/or space? If yes, filters should slide along those dimensions. If no, treat them as "channels". – Dennis Soemers Dec 30 '18 at 20:01
• Thanks again. The idea that the channels are NOT the dimension(s) in which you expect the invariance property makes sense! What happens if there is more than one dimension that fits under this idea of channels? It seems these deep learning libraries only allow for a single input as the "channels." With one dimension, say, of size 5, you would simply set channels = 5. What if you had some data where you had 2 dimensions of size 5 and 10, respectively, in which you are NOT expecting the invariance property, how would you define the number of channels? – Iamanon Dec 30 '18 at 20:37

The concept of channels comes from communications and has little to do with neural networks specifically. We see the concept in AI literature because AI deals with media streams and files.

The concept is this. A media format has channels to allow the independent passage of multiple signals through a single mechanism. That's it.

For example, some streamed broadcast or multicast may have channels for red, green, blue, alpha, left audio, and right audio. In printing, four color is standardized to cyan, magenta, yellow, and black (or key). In jet flight control there are channels for rudder, aileron, elevator, throttle, landing gear, and break positions.

Automated vehicles would have steering, accelerator, break, horn, and two turn signals for control channels and data acquisition channels including an accelerometer, two microphones, probably some strain gauges, four radial encoders, three color channels for each of two cameras, and some thermistors.

In the context of convolution, a kernel may be applied in various ways to individual channels or groups of channels either in parallel or in aggregation. When you turn up the bass on a digital music player, a kernel designed to boost the lower frequencies is applied to both left and right audio channels. Missile countermeasures have kernels with built in hardware trigonometry to track and steer in compound angles with nanosecond response time. Remote control toys have channels through which joystick signals are sent through kernels to coordinate propulsion signals to wheels or props.

For computer vision, consider the five dimensions involved in a data set of 1,000 video examples with resolution of 1,920 x 1,080 progressive 32 bit pixels at 29.97 fps, each example with 9,010 frames.

• Example index
• Frame number
• Vertical index
• Horizontal index
• Channel

This would fit into an array of bytes dimensioned [1,000][9,010][1,080][1,920][4]. How the kernel deals with one or more channels and whether the channels interact or are kept independent has to do with the expected purpose of the layer within the deep learning design, and that is dependent upon what is expected in the video input and what is hoped for as output after the network is trained.

Therefore,

• The AI system topology,
• The configuration and dimensions of layers,
• The dimensions of kernels,
• How those kernels deal with the channels, and
• What criteria are used to determine what is back propagated

is a coordinated set of design decisions the AI engineer must make based on theory and probably demonstrate in a POC later on.

The mapping of 1 channel to grayscale and 3 channels to color is an oversimplification. Sometimes color is RGB at the input and output (3 and 3 channels respectively) but averaged to grays through some layers for the sake of computational thrift whereas a normalized representation of color balance is processed in a separate parallel stream. Other times the alpha channel is needed or an alpha and beta channel, so there may be 4 or 5 channels respectively to handle green or blue screen movie making techniques or to support the learning of scene layering.

The AI engineer must first

• Decide what is wanted,
• What must be learned to get it,
• Design a process to achieve it, and
• Select appropriate learning techniques to tune the parameterized process to learn what is necessary.

After such design activity is reasonably complete, the system topology and the dimension count and sizes become clear. The use of channels to fully represent information and what operations must be performed by kernels on those channels also become apparent.

• Douglas, thanks for the answer. Question pertaining to your 1st sentence about channels not having much relation to neural nets: In deep learning libraries (e.g., Tensorflow, Pytorch), the convolution layer requires the number of channels as an input. Do you know if the meaning for these libraries is the same as the one you described, or is it more arbitrary? I am trying to use CNNs for a computational physics application, described in my other post in datascience.stackexchange.com/questions/43291/… . I may make the same post in the AI board – Iamanon Dec 30 '18 at 19:11
• Ahh I see. This makes sense. Part of my confusion in my specific application is I have not completed the stage of figuring out what exactly I am hoping to accomplish and the full list of input parameters. As you said, once that is done, I should have a much clearer picture of the dimension-related issues. – Iamanon Dec 30 '18 at 21:06

If you have a gray scale image, that means you are getting data from one sensor. If you have an RGB image, that means you are getting data from three sensors. If you have a CMYK image, that means you are getting data from four sensors.

So, channels can be considered as same information seen from different perspective. (here color)

If you see how the kernel (for example 5*5*3) moves, it moves only in XY direction and not in the channel direction. So, you are trying to learn features in XY direction from all the channels together.

But, if you exchange the dimensions like you mentioned your XY dimensions become 200*3 or 3*200 and your channels become 200. In this case you are moving kernel not in the actual XY spatial space of image. So, it doesn't make any sense according to me. You are contradicting the basic concept of CNN by doing so.

The concept of CNN itself is that you want to learn features from the spatial domain of the image which is XY dimension. So, you cannot change dimensions like you mentioned.