How is parameter sharing done in CNN?

Question

I am trying to understand the concept of parameter sharing in a convolution neural network from Parameter Sharing. I have a few confusions:

Parameter sharing refers to the fact that for generating a single activation map, we use the same kernel throughout the image. And for that activation map, the weights of that kernel remain the same through the image?

Denoting a single 2-dimensional slice of depth as a depth slice (e.g. a volume of size [55x55x96] has 96 depth slices, each of size [55x55]), we are going to constrain the neurons in each depth slice to use the same weights and bias.

Does the above paragraph refer to the fact that output of neurons in one activation map is generated by using the same weights in kernel throughout the image? And that kernel is convolved on the entire image?

No. of parameters without parameter sharing: There are 555596 = 290,400 neurons in the first Conv Layer, and each has 11113 = 363 weights and 1 bias. Together, this adds up to 290400 * 364 = 105,705,600 parameters on the first layer of the ConvNet alone. Clearly, this number is very high.

No. of parameters with parameter sharing With parameter sharing scheme, the first Conv Layer in our example would now have only 96 unique set of weights (one for each depth slice), for a total of 9611113 = 34,848 unique weights, or 34,944 parameters (+96 biases). Alternatively, all 5555 neurons in each depth slice will now be using the same parameters. What does this bold sentence mean?

Also, how the parameters are different for both schemes? In both cases, we are using 96 kernels with 11113 size and the resulting output is 55*55. Then how the number of parameters for both schemes coming out to be different?

Answer 1

Concerning parameter sharing.

1. For the fully connected neural network you have an input of shape (H_in * W_in * C_in) and the output of shape (H_out * W_out * C_out). This means, that each color of the pixel of the output feature map is connected to every color of the pixel from the input feature map. There is a separate learnable parameter for each pixel in the input image and the output. Hence, one gets this huge number of parameters : (H_in * H_out * W_in * W_out * C_in * C_out)
2. In the convolutional layer the input is the image of shape (H_in, W_in, C_in) and the weights account for the neighborhood of the given pixel, say of size K * K. The output is obtained as a weighted sum of the given pixel and its neighborhood. There is a separate kernel for each pair of the input and output channel (C_in, C_out), but the weights of the kernel (a tensor of shape (K, K, C_in, C_out) are independent of the location. Actually, this layer can accept images of any resolution, whereas, the fully connected can work only with a fixed resolution. Finally one has (K, K, C_in, C_out) parameters, which for the kernel size K much smaller, than the input resolution result into significant drop in the number of variables.