What is a convolutional neural network?

Question

Given that this question has not yet been asked on this site, although similar questions have already been asked in the past (e.g. here or here), what is essentially a convolutional neural network (CNN)? Why are they heavily used in image recognition applications of machine learning?

Answer 1

(Of course, similar questions have been asked in the past and there are many sites, papers, video lessons, online that explain how CNNs work, but I think it's still a good idea to have a reference answer that hopefully will give you the main ideas behind CNNs.)

A convolutional neural network (CNN) is a neural network that performs the convolution (or cross-correlation) operation typically followed by some downsampling (aka pooling) operations.

The convolution operation comes from the mathematical equivalent operation, which is an operation that takes as inputs two functions $f$ and $h$ and produces another function $g$ as output, which is often denoted as $f \circledast h = g$, where $\circledast$ is the convolution operation (or operator).

In image processing, the convolution is used to process images in multiples ways. For example, it is used to remove noise (e.g. the convolution of a noisy image with a Gaussian kernel produces a smoother image) or to compute derivatives of the image, which can then be used e.g. to detect edges or corners, which are usually the main features of an image. For example, the Harris corner detector makes use of the partial derivatives (in the $x$ and $y$ direction) of the image to find corners (or interest points) in the image.

In the context of CNNs, $f$ is the image (in the case of the first layer of the CNN) or a so-called feature map (in the case of hidden layers), $h$ is the kernel (aka filter) and $g$ is also a feature map. (In this answer, I explain these concepts, including how an image can be viewed as a function, more in detail, so I suggest that you read it).

Here's an illustrative example of how the convolution works.

where the $\color{blue}{\text{blue}}$ grid is the image, the $\color{gray}{\text{gray}}$ grid is the kernel and $\color{green}{\text{green}}$ grid is the feature map (i.e. the output of the convolution between the image and the kernel). The white squares are around the images represent the padding that is added around the image so that the convolution operation produces a feature map of a specific dimension.

Essentially, the convolution is a series of dot products between the kernel and different patches (or parts) of the image. The convolution can even be represented as matrix multiplication, so the name convolution shouldn't scare you anymore, if you are familiar with dot products and matrix multiplications.

As opposed to many operations in image processing where the kernels are typically fixed, in the context of CNNs, the kernels are learnable parameters, i.e. they change depending on the loss function, so they are supposed to represent functions that when convolved with their respective input functions are useful to extract meaningful information (i.e. features) to solve the task the CNN is being trained to solve. For this reason, the convolution is often thought of as an operation that extracts features from images. In fact, the output of the convolution, in the context of CNNs, is often called feature map. Moreover, a CNN is typically thought of as a data-driven feature extractor for the same reason.

There are many different variations of the standard convolution operation. For example, there are transposed convolutions, dilated convolutions, etc., which are used to solve slightly different problems. For example, the dilated convolution can be used to increase the receptive field of an element of a feature map (which is typically a desirable property for several reasons). There are also upsampling operations which are particularly useful in the context of image segmentation. These different convolutions and their arithmetic are explained very well in the paper A guide to convolution arithmetic for deep learning by Vincent Dumoulin and Francesco Visin. Here's also the associated Github repository that contains all the images in this paper that show how the convolution operations work (from which I also took the gif above).

To conclude, CNNs are very useful to process images and extract features from them because they use convolution operations (and downsampling and upsampling operations). They can be used for image (or object) classification, object detection (i.e. object localization with a bounding box + object classification), image segmentation (including semantic segmentation and instance segmentation), and possibly many other tasks where you need to learn a function that takes as input images and needs to extract information from those images to get someone high-level (but also low-level) output (e.g. the name of the object in the image), given some training data.