Are filters fixed or learned?

Question

No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filters' weights are learned by means of back-prop.

But how are kernels themselves selected? I know kernels had been used in image processing long before CNNs, hence I'd imagine there would be a set of filters based on kernels (see, for example, this article) that are proven to be effective for edge detection and the likes.

Reading around the web, I also found something about "randomly generated kernels". Does anyone know if and when this practice is adopted?

Answer 1

What are filters in image processing?

In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the image. For example, you can use a Gaussian filter to smooth the image (including its edges).

What are filters in CNNs?

In the context of convolutional neural networks (CNNs), the filters (or kernels) are the learnable parameters of the model.

Before training, the kernels are usually randomly initialised (so they are not usually hardcoded). During training, depending on the loss or error of the network (according to the loss function), the kernels (or filters) are updated, so that to minimise the loss (or error). After training, they are typically fixed. Incredibly, the filters learned by CNNs can be similar to the Gabor filter (which is thought to be related to our visual system [1]). See figure 9.19 of chapter 9 (p. 365) of the Deep Learning book by Goodfellow et al.

The number of kernels that are applied to a given input (and more than one kernel is often applied) in a CNN is a hyper-parameter.

What are the differences and similarities?

In both contexts, the words "kernel" and "filter" are roughly synonymous, so they are often used interchangeably. Furthermore, in both cases, the kernels are related to the convolution (or cross-correlation) operation. More specifically, the application of a filter, which is a function $h$, to an input, which is another function $f$, is equivalent to the convolution of $f$ and $h$. In mathematics, this is often denoted by $f \circledast h = g$, where $\circledast$ is the convolution operator and $g$ is the result of the convolution operation and is often called the convolution (of $f$ and $h$). In the case of image processing, $g$ is the filtered image. In the case of CNNs, $g$ is often called an activation map.

Further reading

Take a look at this and this answers for more details about CNNs and the convolution operation, respectively.

You may also want to have a look at Visualizing what ConvNets learn for some info about the visualization of the kernels learned during the training of a CNN.

Answer 2

If you're looking for filters with known effect, the Gaussian filters do smoothing, the Gabor filters are useful for edge detection, etc.

Usually, in deep learning models where things are trained from scratch, the filters are randomly initialized and then learned by the model's training scheme. For the most part, without using any of the well-known kernels mentioned above.

Clarification on for the most part: so filters aren't initialized with the goal of knowing exactly which feature it will activate on, but what will assist the training procedure. Recently, people have even trained ResNets with or without batch normalization by finding good initial points -- it's an ongoing field of research.