Why is the number of output channels 16 in the hidden layer of this CNN?

Question

In this tutorial from Jeremy Howard: What is torch.nn really? he has an example towards the end where he creates a CNN for MNIST. In nn.Conv2d, he makes the in_channels and out_channels: (1,16), (16,16), (16,10).

I get that the last one has to be 10 because there are 10 classes and we want 'probabilities' of each class. But why go up to 16 first? How do you choose this value? And why not just go from 1 to 10, 10 to 10, and 10 to 10? Does this have to do with the kernel_size and stride?

All of the images are 28x28, so I can't see any correlation between these values and 16 either.

class Mnist_CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1)

    def forward(self, xb):
        xb = xb.view(-1, 1, 28, 28)
        xb = F.relu(self.conv1(xb))
        xb = F.relu(self.conv2(xb))
        xb = F.relu(self.conv3(xb))
        xb = F.avg_pool2d(xb, 4)
        return xb.view(-1, xb.size(1))

Answer 1

I understand your question as: "How did the author select the number of neurons in their hidden layer?"

The number of neurons in the hidden layer is how you can control the complexity of the function you are trying to generate to map the inputs to an output. The more neurons in the hidden layer the more complex the function thus you can capture more intricate decision barriers. However, the more complex function is harder to optimize and will lead to lower performance scores. The goal here is to find the right tradeoff to maximize your performance. You can tune the number of hidden neurons as a hyper-parameter using cross-validation.

There isn't any formula to determine the number of neurons you will need, however you can get an intuition based on the number of inputs and outputs you will have. Generally, you want more hidden neurons than input and output neurons. Since most people are programmers who are writing neural networks, we are used to working with units in $2^n$. Thus, 16 is chosen over 10, and 32 would be chosen over 28.