What gets optimized in convolutional neural network?

Question

In a convolutional neural network, the hyperparameters such as number of kernels and stride, kernel size, etc are determined. After some combination of convolutions, ReLU and pooling layer there is the fully connected (FC) layer in the end which yields a classification result. I originally thought that during training the values of kernels would be optimized and that kernels such as edge detection are a result of optimization.

But at the end if we have weights to optimize at the FC layer, what is it that gets optimized during training of the CNN? Do both the kernel values and weights in FCC get optimized? If so, it seems like we're dealing with two different types of parameters. How are both trained simultaneously? If not so, are there simply sets of kernels known to work and automatically implemented in CNN modules?

Answer 1

Do both the kernel values and weights in FCC get optimized?

Yes.

Some of the designs for image processing neural networks prior to CNNs had separate filter processing states. For instance, Sobel filters were popular choices in earlier attempts at machine learning on images, and they can be thought of as fixed CNN-like layers. They may still have a role in some projects.

However, most CNN architectures for images now work with pixel data directly, and can learn both the filter weights and fully connected later weights together.

In some uses, such as transfer learning, it is useful to be able to selectively learn only some of the layers. You may take the CNN filter layers from a very general image classifier trained on ImageNet, and repurpose it by replacing the fully connected layers. When training the new neural network you can freeze the filter layers and learn only the fully connected layers - although there is no specific requirement to separate them only by convolutional/fully-connected, you could equally retrain only part of the fully-connnected layers, or include some of the convolutional layers.

If so, it seems like we're dealing with two different types of parameters. How are both trained simultaneously?

They are not really that different. A CNN can be thought of as fully connected throughout, but with some extra constraints on the convolutional layers:

• Weights that would connect a feature map neuron to an input outside the bounds of the filter are always zero.

• Weights for each position on the feature map are identical.

Using learnable convolution filters enforces both these constraints.

The practical difference to how back propagation works in convolutional layers as opposed to fully connected layers is to sum all gradients arising from each "pixel" in the feature map to the appropriate filter weight when back propagating. So unlike a fully connected layer weight that receives one summed gradient update from the layer above, each convolutional filter weight receives the equivalent update summed over all pixels in the next feature layer. Depending on which source you learn CNNs from, this additional outer sum might be shown or might be implied using different notation for the update rules.