How is the convolution operation connected to neural networks?

Question

I've been reading up on the convolution operation and neural networks. I understand that the convolution operation is defined as:

$$(f * g)(t)=\int_{-\infty}^{\infty} f(\tau) g(t-\tau) d \tau$$

The convolution operation has some properties, such as commutativity, associativity, etc.

How is the convolution connected to neural networks? How do we use this operation in a CNN?

Answer 1

The convolution operation performed by most CNNs that you will find (on the web) assumes that the signals/functions are discrete and 2-dimensional (e.g. images can be viewed as 2-dimensional discrete signals), although this does not have to be the case. In fact, 1D and 3D convolutions are also implemented in several deep learning libraries (see here for an example).

The equation in your post describes the convolution operation for 1-dimensional continuous signals with domain $\mathbb{R}$ (sometimes called the time domain, hence the usage of the letter $t$), so we use an integral that goes from $-\infty$ to $\infty$.

An equation that better describes the common 2D convolution operation performed by CNNs is the following

$$S(i, j)=(K * I)(i, j)=\sum_{m=-M}^M \sum_{n=-N}^N I(i+m, j+n) K(m, n), \tag{1}\label{1}$$

where

• $S(i, j)$ is the convolution of the signals $I$ (e.g. an image) and $K$ (e.g. the kernel) evaluated at the coordinates (or points of the domain) $(i, j)$, so $S$ is the function that results from the convolution of $I$ and $K$
• $(K * I)(i, j)$ is just another notation for $S(i, j)$, which emphasizes that the convolution is performed between $K$ and $I$, and $*$ is the convolution operation
• $K(m, n)$ is the value of the function $K$ (in the context of neural networks and image processing, this is often known as the kernel, hence the $K$, or filter) evaluated at $(m, n)$
• similarly, $I(i+m, j+n)$ is the value of the function $I$ (e.g. the image) evaluated at $(i+m, j+n)$

So, essentially, equation \ref{1} is telling us that the value of the convolution between $I$ and $K$, denoted by $S$, evaluated at $(i, j)$ is a sum of products, where the factors are $I(i+m, j+n)$ and $K(m, n)$, where the values of $m$ and $n$ depend on $M$ and $N$, which could also be equal, and typical values would be, for example, $1$ or $2$, so equation \ref{1} could be written as follows

$$S(i, j)=(K * I)(i, j)=\sum_{m=-1}^1 \sum_{n=-1}^1 I(i+m, j+n) K(m, n), \tag{2}\label{2}$$

If you expand the summations, you will get

$$S(i, j)= I(i-1, j-1) K(-1, -1) + I(i-1, j) K(-1, 0) + \dots + I(i+1, j+1) K(1, 1)$$

This is essentially a 2D dot product or, equivalently, an element-wise multiplication followed by the addition of those multiplications.

Another important thing to note is that the convolution of $I$ and $K$ evaluated at $(i, j)$ uses the values of $I$ with respect to the points $(i, j)$.

Moreover, in a 2D convolutional layer of a CNN, we do not just compute $S(i, j)$, but we compute $S(i, j)$ for all possible values of $i$ and $j$, which are typically the coordinates of the image (in the case of the first layer).

Finally, the equation \ref{2} actually describes what is known as the cross-correlation operation, which is exactly the same as the convolution operation, but we use $+$ instead of $-$ in $I(i+m, j+n)$. In practice, in the context of CNNs, it doesn't really matter whether you use the $+$ and $-$, given that the kernels $K$ are learned. However, the convolution operation and the cross-correlation would be different in the case the kernels are not symmetric, but identical in the case they are. You can read more about this topic in this answer that I had written a while ago.

You should probably also read this chapter that more thoroughly describes all these operations. There are also other answers that I had written in the past that could be useful, such as this, this, this, this, and/or this. By the time you have understood all these answers, you will also have understood the convolution operation and how it is related to CNNs.