Is it possible to have different channel dimensions in a CNN?

Question

Let's say I have two channels that I wish to feed into a CNN. One of the channel contains 4 traces and has a width of 512. Stacking them on top of each other therefore yields an image with dimensions (4, 512). The other channel is just 1 trace, so its dimensions would be (1, 512).

I then have convolutional filters that are of dimension (1, 5) as an example. That means that the filters run over each trace separately. The first channel (containing the 4 traces) will then have a set of filter weights, shared among the 4 traces. The second channel (containing the 1 trace) will have a completely different set of weights (as per this SE question).

TLDR: Can convolutional layers in a CNN have different dimensions? Putting this in the context of images: Could we have a CNN that takes an image that has dimensions (100, 100) for the red channel, (100, 100) for the green channel, and (50, 100) for the blue channel?

Answer 1

You can do whatever the heck you want.

Of course you will have to design the data flow through the network so that it can make whatever inferences you intend it to make.

The first channel (containing the 4 traces) will then have a set of filter weights, shared among the 4 traces. The second channel (containing the 1 trace) will have a completely different set of weights (as per this SE question).

Sure, you can do that. No reason why you couldn't. Will it work well? Who knows. Have to try it and see.

The best way to combine them will depend on what the NN is supposed to actually do. With the architecture you've described, at least this layer is unable to relate traces to each other. This would be bad when processing, for example, colour images - you don't want to treat red, green and blue the same way as each other, and you want to detect certain combinations of red, green and blue. If you also want the network to have this ability, then maybe you should treat each trace as a channel so the network can see all of them at once.

At some point you will obviously have to combine the results together.

Could we have a CNN that takes an image that has dimensions (100, 100) for the red channel, (100, 100) for the green channel, and (50, 100) for the blue channel?

As I said, it depends on what the network is supposed to do. Are these channels totally separate? Then you can process them with different sized CNNs - or even the same CNN until the dense layers - and combine the results in the dense layers at the end. But if these are the RGB components of one image, you'd be better off just stretching the blue channel so the CNN can recognize colours like yellow.