How is the depth of a convolutional layer determined?

Question

I am looking at a diagram of ZFNet below, in an attempt to understand how CNNs are designed.

In the first layer, I understand the depth of 3 (224x224x3) is the number of color channels in the image.

In the second layer, I understand the $110 \times 110$ is computed with the formula $W_2 = (W_1-F+2P)/S + 1 = (224 - 7 + 2*1/2)/2 + 1 = 110$.

I also understand how pooling works to create a size reduction from $110 \times 110$ to $55 \times 55$.

But where does the depth of $96$ come from in the second layer? Is this the new "batch size"? Is it totally arbitrary?

Bonus points if someone can direct me to a reference that can help me understand how all these dimensions relate to each other.

Answer 1

The $96$ is the number of feature maps, which is equal to the number of filters/kernels.

The choice of the number of kernels is not fully arbitrary, although there is no equation or exact rule restricting the number.

If you have a CNN, one single convolution operation would be pointless: since it is used for the whole image information, it can generalize, but only to specific (meaning: a finite amount of) features. Easy example: if a 7x7 filter in the first layer concentrates on round shapes, it can not generalize on let's say red cubes at the same time.

Therefore, your convolutional layers have several filters/kernels, i.e. several weights where each is used for a convolution. The result of each of these convolutions is one filter/feature map, i.e. the image information convolved by a kernel.

Typically, you look at your kernels and your problem's domain to figure out what an appropriate number of kernels could be. You also have to keep in mind that too few kernels could possibly lose information and overfit specific patterns, while too many kernels could possibly underfit. Especially, when you have far more parameters (primarily the weights of your neural network) than training data, your neural network will normally perform bad and you have to reduce its size.

The kernels should not be confused with batch size. The batch size is the number of samples (here images) you train in parallel. Each training step of your neural network does not only consist of feeding a single image, but feeding batch size number of images, usually combined with batch normalization steps between the layers. Hence, this has nothing to do with the number of kernels.

Answer 2

Let's say you have an image with $3$ channels and you have $10$ filters, where each filter has the shape $5 \times 5 \times 3$. The depth of the convolutional layer after having applied this filter to the image is $10$, which is equal to the number of filters. The spatial dimensions of the filter (in this case, $5 \times 5$) are more or less defined arbitrarily (it's a hyperparameter).

Answer 3

Given that you were also asking for a reference that describes in detail these operations, you should take a look at the paper A guide to convolution arithmetic for deep learning (2018), which describes in detail the arithmetic of many convolution operations used in convolutional neural networks. There's also the associated repo with the animations of the different operations.