Let's assume an extreme case in which the kernel of the convolution layer takes only values 0 or 1. To capture all possible patterns in input of $C$ number of channels, we need $2^{C*K_H*K_W}$ filters, where $(K_H, K_W)$ is the shape of a kernel. So to process a standard RGB image with 3 input channels with 3x3 kernel, we need our layer to output $2^{27}$ channels. Do I correctly conclude that according to this, the standard layers of 64 to 1024 filters are only able to catch a small part of (perhaps) useful patterns?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ what is the point of assuming that extreme case? Is it only for the sake of asking or you have a practical task where you are restricted to use only 0 and 1 filters? $\endgroup$– bit_scientistCommented Jan 2, 2020 at 3:50
-
1$\begingroup$ @voo_doo Maybe instead of extreme, I should use the word edge or least complex. The more values a kernel can take, the higher the power base and the more filters are needed to capture all the patterns. So I wanted to say that the layer with the continuous kernel should use at least more filters than the layer with the binary kernel. This question came to me when I read about wide_resnet, which improved the results by increasing the number of filters in layers. $\endgroup$– KasiaCommented Jan 2, 2020 at 7:06
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
From mathematical point of view you are correct as are your calculations. To catch all the patterns you need that many filters, but this is where a whole idea of a training comes in. Main objective of the training in the CNNs is to find just a few good patterns from billions possible ones.
So the direct answer to your question is: The standard layers of 64 to 1024 filters are only able to catch a small part of (perhaps) useful patterns, yes but this is assuming no training taking place. If you conducted training on given data with given model, then 64 to 1024 filters could already extract a lot of useful patterns, perhaps more than needed.
$\begingroup$
$\endgroup$
4
Let $n=C*K_w*K_h$. Then you should only need $n$ filters. Not $2^n$ to keep all the information. If you just used the rows of the identity matrix as your filters than your convolution would just be making an exact copy so it definitely wouldn't be throwing away information. On the other hand, there will be a max pooling operation. To simplify the question let's suppose we have a 3 channels, and a 1 by 1 kernel. And then let's suppose it is just one convolution followed by global max pooling. Also, let's use your assumption that it's all binary. If you have $m$ filters then the final output will be $m$ dimensional no matter how many input points you have. So clearly information is being thrown away there. But that's not such a bad thing. Throwing away irrelevant information gets us closer to the features we need the problem at hand. The parts that get thrown away by max pooling correspond to features not being found in a particular part of the image.
-
$\begingroup$ Keeping information is not the goal here. The layer is supposed to add new information's about the relationship between the data in the input. To find all possible relationships between just two RGB pixels using binary kernel, we need $2^{2*3}=64$ filters. Dropping some of the information's may not be the problem in later layers working on high level features, but the first layers need to find a kernel, that basically fits all cases, because this kernel has to find some useful pattern in every single slice of the image. $\endgroup$– KasiaCommented Jan 2, 2020 at 23:56
-
$\begingroup$ If you take just 6 filters and plug the output of your convolution into a dense MLP then it should be able to approximate any function of your data assuming you have enough neurons. But if you first put it through maxpooling as is normally done then having 64 filters would let you extract every possible feature in the output of your maxpooling. But you have still lost information by doing max pooling. So using 64 filters would with maxpooling would let you approximate any function but only as long as you don't care about duplicates in the input $\endgroup$– jgleoj23Commented Jan 3, 2020 at 1:44
-
$\begingroup$ I'm a little lost in what you talk about. Why do you keep referring to using max pooling? The current trend is to move away from using it and replace it with strided convolutions. And I've never seen it used in front of a convolutions layer. I'm trying hard to understand what you're trying to explain to me, but it' s still eluding me. $\endgroup$– KasiaCommented Jan 3, 2020 at 11:30
-
$\begingroup$ i'm saying that what you said is true in the case where we use max pooling. Of course, 2^29 is way more than we'll need or be able to use anytime soon. $\endgroup$– jgleoj23Commented Jan 4, 2020 at 0:17