I am trying to understand the concept of parameter sharing in a convolution neural network from Parameter Sharing. I have a few confusions:
Parameter sharing refers to the fact that for generating a single activation map, we use the same kernel throughout the image. And for that activation map, the weights of that kernel remain the same through the image?
Denoting a single 2-dimensional slice of depth as a depth slice (e.g. a volume of size [55x55x96] has 96 depth slices, each of size [55x55]), we are going to constrain the neurons in each depth slice to use the same weights and bias.
Does the above paragraph refer to the fact that output of neurons in one activation map is generated by using the same weights in kernel throughout the image? And that kernel is convolved on the entire image?
No. of parameters without parameter sharing: There are 555596 = 290,400 neurons in the first Conv Layer, and each has 11113 = 363 weights and 1 bias. Together, this adds up to 290400 * 364 = 105,705,600 parameters on the first layer of the ConvNet alone. Clearly, this number is very high.
No. of parameters with parameter sharing With parameter sharing scheme, the first Conv Layer in our example would now have only 96 unique set of weights (one for each depth slice), for a total of 9611113 = 34,848 unique weights, or 34,944 parameters (+96 biases). Alternatively, all 5555 neurons in each depth slice will now be using the same parameters. What does this bold sentence mean?
Also, how the parameters are different for both schemes? In both cases, we are using 96 kernels with 11113 size and the resulting output is 55*55. Then how the number of parameters for both schemes coming out to be different?