2
$\begingroup$

Let's assume an extreme case in which the kernel of the convolution layer takes only values 0 or 1. To capture all possible patterns in input of $C$ number of channels, we need $2^{C*K_H*K_W}$ filters, where $(K_H, K_W)$ is the shape of a kernel. So to process a standard RGB image with 3 input channels with 3x3 kernel, we need our layer to output $2^{27}$ channels. Do I correctly conclude that according to this, the standard layers of 64 to 1024 filters are only able to catch a small part of (perhaps) useful patterns?

$\endgroup$
  • $\begingroup$ what is the point of assuming that extreme case? Is it only for the sake of asking or you have a practical task where you are restricted to use only 0 and 1 filters? $\endgroup$ – bit_scientist Jan 2 at 3:50
  • 1
    $\begingroup$ @voo_doo Maybe instead of extreme, I should use the word edge or least complex. The more values a kernel can take, the higher the power base and the more filters are needed to capture all the patterns. So I wanted to say that the layer with the continuous kernel should use at least more filters than the layer with the binary kernel. This question came to me when I read about wide_resnet, which improved the results by increasing the number of filters in layers. $\endgroup$ – Kasia Jan 2 at 7:06
1
$\begingroup$

From mathematical point of view you are correct as are your calculations. To catch all the patterns you need that many filters, but this is where a whole idea of a training comes in. Main objective of the training in the CNNs is to find just a few good patterns from billions possible ones.

So the direct answer to your question is: The standard layers of 64 to 1024 filters are only able to catch a small part of (perhaps) useful patterns, yes but this is assuming no training taking place. If you conducted training on given data with given model, then 64 to 1024 filters could already extract a lot of useful patterns, perhaps more than needed.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Let $n=C*K_w*K_h$. Then you should only need $n$ filters. Not $2^n$ to keep all the information. If you just used the rows of the identity matrix as your filters than your convolution would just be making an exact copy so it definitely wouldn't be throwing away information. On the other hand, there will be a max pooling operation. To simplify the question let's suppose we have a 3 channels, and a 1 by 1 kernel. And then let's suppose it is just one convolution followed by global max pooling. Also, let's use your assumption that it's all binary. If you have $m$ filters then the final output will be $m$ dimensional no matter how many input points you have. So clearly information is being thrown away there. But that's not such a bad thing. Throwing away irrelevant information gets us closer to the features we need the problem at hand. The parts that get thrown away by max pooling correspond to features not being found in a particular part of the image.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Keeping information is not the goal here. The layer is supposed to add new information's about the relationship between the data in the input. To find all possible relationships between just two RGB pixels using binary kernel, we need $2^{2*3}=64$ filters. Dropping some of the information's may not be the problem in later layers working on high level features, but the first layers need to find a kernel, that basically fits all cases, because this kernel has to find some useful pattern in every single slice of the image. $\endgroup$ – Kasia Jan 2 at 23:56
  • $\begingroup$ If you take just 6 filters and plug the output of your convolution into a dense MLP then it should be able to approximate any function of your data assuming you have enough neurons. But if you first put it through maxpooling as is normally done then having 64 filters would let you extract every possible feature in the output of your maxpooling. But you have still lost information by doing max pooling. So using 64 filters would with maxpooling would let you approximate any function but only as long as you don't care about duplicates in the input $\endgroup$ – jgleoj23 Jan 3 at 1:44
  • $\begingroup$ I'm a little lost in what you talk about. Why do you keep referring to using max pooling? The current trend is to move away from using it and replace it with strided convolutions. And I've never seen it used in front of a convolutions layer. I'm trying hard to understand what you're trying to explain to me, but it' s still eluding me. $\endgroup$ – Kasia Jan 3 at 11:30
  • $\begingroup$ i'm saying that what you said is true in the case where we use max pooling. Of course, 2^29 is way more than we'll need or be able to use anytime soon. $\endgroup$ – jgleoj23 Jan 4 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.