2
$\begingroup$

In this tutorial from Jeremy Howard: What is torch.nn really? he has an example towards the end where he creates a CNN for mnist. In nn.Conv2d he makes the inchannels and outchannels: (1,16), (16,16), (16,10). I get that the last one has to be 10 because there are 10 classes and we want 'probabilities' of each class. But why go up to 16 first? How do you choose this value? And why not just go from 1 to 10, 10 to 10, and 10 to 10? Does this have to do with the kernel_size and stride?

All of the images are 28x28 so I can't see any correlation between these values and 16 either.

class Mnist_CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1)

    def forward(self, xb):
        xb = xb.view(-1, 1, 28, 28)
        xb = F.relu(self.conv1(xb))
        xb = F.relu(self.conv2(xb))
        xb = F.relu(self.conv3(xb))
        xb = F.avg_pool2d(xb, 4)
        return xb.view(-1, xb.size(1))
```
$\endgroup$
2
$\begingroup$

I understand your question as: "How did the author select the number of neurons in their hidden layer?"

The number of neurons in the hidden layer is how you can control the complexity of the function you are trying to generate to map the inputs to an output. The more neurons in the hidden layer the more complex the function thus you can capture more intricate decision barriers. However, the more complex function is harder to optimize and will lead to lower performance scores. The goal here is to find the right tradeoff to maximize your performance. You can tune the number of hidden neurons as a hyper-parameter using cross-validation.

There isn't any formula to determine the number of neurons you will need, however you can get an intuition based on the number of inputs and outputs you will have. Generally, you want more hidden neurons than input and output neurons. Since most people are programmers who are writing neural networks, we are used to working with units in $2^n$. Thus, 16 is chosen over 10, and 32 would be chosen over 28.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.