# How many parameters are being optimised over in a simple CNN?

Okay so here's my CNN (simple example from a tutorial) along with some arithmetic to get the total number of free parameters.

We've got a dataset of 28*28 grayscale image (MNIST).

1. First layer is a 2D convolution using 32 3x3 kernels. Dimensionality of the output is 26x26x32 (kernel stride length was 1 and we have 32 feature maps of 26x26). Running parameter count: 288
2. Second layer is 2x2 MaxPool with a 2x2. Dimensionality of the output is 13x13x32 but then we flatten so we got a vector of length 5408. No extra parameters here.
3. Third layer is Dense. A 5408x100 matrix. Dimensionality of the output is 100. Running Parameter count: 540988
4. Fourth layer is Dense also. A 100x10 matrix. Dimensionality of the output is 10. Running Parameter count: 541988

Then we're supposed to do stochastic gradient descent on a 541988 parameter space!

That feels like a ridiculously big number to me. And this is meant to be the hello world problem of CNNs. Am I missing something fundamental in my understanding of how this is meant to work? Or maybe the number is correct but it's not actually a big deal for a computer to crunch?

In case it helps. Here is how the model was built in Keras:

def define_model():
model = Sequential()
model.add(Conv2D(32, (3,3), activation = 'relu', kernel_initializer = 'he_uniform', input_shape=(28,28,1)))
opt = SGD(lr=0.01, momentum=0.9)
model.compile(optimizer=opt, loss='categorical_crossentropy', metric=['accuracy'])
return model

• I do not think the number is that high, but you can always lower it by making 1 more convolution early for example – Miguel Saraiva Dec 26 '19 at 19:33
• Is there something wrong in my working? Also, wouldn't adding another convolution simply increase the number of free parameters? – Alexander Soare Dec 26 '19 at 19:40
• Depends on the kernel size, if you use a bigger one without padding no. I do not see anything wrong with it but I'm no expert. If you are worried about the number of parameters probably adding another pair Conv2D + Pool(2,2) will reduce them. Although it adds a new layer with parameters it will reduce the size of ur current third layer by a lot in terms of number of parameters. – Miguel Saraiva Dec 26 '19 at 20:01
• Thanks @MiguelSaraiva. I'm not so worried about reducing the dimensionality. I was more interested in consolidating my understanding of the optimisation problem. I think it's about right. I used print(model.count_params()) and got 542230. I just need to figure out where the extra 242 parameters came from. – Alexander Soare Dec 26 '19 at 20:12

Neural networks can have a lot of different structures. CNNs can have a number of parameters that ranges from a few thousands to several millions.

In general you aim to increase the number of filters and reduce the first 2 dimensions, as you go deeper in the network.

So if you had Conv -> pool -> Conv -> pool -> ... , you could do for example first conv with kernel size = 5 and 8 filters and the second conv with kernel size = 5 and 16 filters. And both pools being (2,2).. But this is just an example.

In your network you start with a 28*28 image, and you use 32 3*3 filters. so number of parameters is (3*3 + 1) * 32 = 320.

In the dense layer you have as input a 13*13*32 and use a 100 FC layer. so n_parameters is (13*13*32 + 1)*100 which is 540900.

Then you get (100+1) * 10 FC which is 1010 more.

total = 320 + 540900 + 1000 which is 542230, as expected.

The +1 that shows up in every layer is the Bias neuron. Basically you add a bias neuron per output in a connection between 2 layers. In the FC1000 to the FC10 it is easy to understand, you have a bias per output neuron. In the Convolutional layers you have a bias term per each of the filters applied, so for each filter you have the filters weights plus 1 for the bias.

Apart from that you also had a small math mistake when adding the 288 at the beginning, you added 188. So your missing 242 parameters were: 32 from the conv layer, 100 from the first dense, 10 from the second dense and 100 from the sum. 100+100+32+10 = 242.