1
$\begingroup$

Okay so here's my CNN (simple example from a tutorial) along with some arithmetic to get the total number of free parameters.

We've got a dataset of 28*28 grayscale image (MNIST).

  1. First layer is a 2D convolution using 32 3x3 kernels. Dimensionality of the output is 26x26x32 (kernel stride length was 1 and we have 32 feature maps of 26x26). Running parameter count: 288
  2. Second layer is 2x2 MaxPool with a 2x2. Dimensionality of the output is 13x13x32 but then we flatten so we got a vector of length 5408. No extra parameters here.
  3. Third layer is Dense. A 5408x100 matrix. Dimensionality of the output is 100. Running Parameter count: 540988
  4. Fourth layer is Dense also. A 100x10 matrix. Dimensionality of the output is 10. Running Parameter count: 541988

Then we're supposed to do stochastic gradient descent on a 541988 parameter space!

That feels like a ridiculously big number to me. And this is meant to be the hello world problem of CNNs. Am I missing something fundamental in my understanding of how this is meant to work? Or maybe the number is correct but it's not actually a big deal for a computer to crunch?

In case it helps. Here is how the model was built in Keras:

def define_model():
    model = Sequential()
    model.add(Conv2D(32, (3,3), activation = 'relu', kernel_initializer = 'he_uniform', input_shape=(28,28,1)))
    model.add(MaxPooling2D((2,2)))
    model.add(Flatten())
    model.add(Dense(100, activation='relu', kernel_initializer='he_uniform'))
    model.add(Dense(10, activation='softmax'))
    opt = SGD(lr=0.01, momentum=0.9)
    model.compile(optimizer=opt, loss='categorical_crossentropy', metric=['accuracy'])
    return model
$\endgroup$
  • $\begingroup$ I do not think the number is that high, but you can always lower it by making 1 more convolution early for example $\endgroup$ – Miguel Saraiva Dec 26 '19 at 19:33
  • $\begingroup$ Is there something wrong in my working? Also, wouldn't adding another convolution simply increase the number of free parameters? $\endgroup$ – Alexander Soare Dec 26 '19 at 19:40
  • $\begingroup$ Depends on the kernel size, if you use a bigger one without padding no. I do not see anything wrong with it but I'm no expert. If you are worried about the number of parameters probably adding another pair Conv2D + Pool(2,2) will reduce them. Although it adds a new layer with parameters it will reduce the size of ur current third layer by a lot in terms of number of parameters. $\endgroup$ – Miguel Saraiva Dec 26 '19 at 20:01
  • $\begingroup$ Thanks @MiguelSaraiva. I'm not so worried about reducing the dimensionality. I was more interested in consolidating my understanding of the optimisation problem. I think it's about right. I used print(model.count_params()) and got 542230. I just need to figure out where the extra 242 parameters came from. $\endgroup$ – Alexander Soare Dec 26 '19 at 20:12
2
$\begingroup$

Neural networks can have a lot of different structures. CNNs can have a number of parameters that ranges from a few thousands to several millions.

In general you aim to increase the number of filters and reduce the first 2 dimensions, as you go deeper in the network.

So if you had Conv -> pool -> Conv -> pool -> ... , you could do for example first conv with kernel size = 5 and 8 filters and the second conv with kernel size = 5 and 16 filters. And both pools being (2,2).. But this is just an example.

In your network you start with a 28*28 image, and you use 32 3*3 filters. so number of parameters is (3*3 + 1) * 32 = 320.

In the dense layer you have as input a 13*13*32 and use a 100 FC layer. so n_parameters is (13*13*32 + 1)*100 which is 540900.

Then you get (100+1) * 10 FC which is 1010 more.

total = 320 + 540900 + 1000 which is 542230, as expected.

The +1 that shows up in every layer is the Bias neuron. Basically you add a bias neuron per output in a connection between 2 layers. In the FC1000 to the FC10 it is easy to understand, you have a bias per output neuron. In the Convolutional layers you have a bias term per each of the filters applied, so for each filter you have the filters weights plus 1 for the bias.

Apart from that you also had a small math mistake when adding the 288 at the beginning, you added 188. So your missing 242 parameters were: 32 from the conv layer, 100 from the first dense, 10 from the second dense and 100 from the sum. 100+100+32+10 = 242.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.