3
$\begingroup$

No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filter weights are learned by means of back-prop.

But how are kernels themselves selected? I know kernels had been used in image processing long before convNets, hence I'd imagine there would be a set of filters based on kernels (e.g.) that are proven to be effective for edge detection and the likes.

Reading around the web, I also found something about "randomly generated kernels". Does anyone know if and when this practice is adopted?

Thanks in advance for any answer!

$\endgroup$
3
$\begingroup$

In the context of image or, in general, signal processing, the kernels (or filters) are used to perform some specific operation on the image. For example, you can use a Gaussian filter to smooth the image (including its edges).

In the context of convolutional neural networks (CNNs), the filters (or kernels) are the learnable parameters of the model. Before training, the kernels are usually randomly initialised (so they are not "selected"). During training, depending on the loss or error of the network (according to the loss function), the kernels (or filters) are updated, so that to minimise the loss (or error). After training, they are fixed. The number of kernels that are applied to a given input (and more than one kernel is often applied) in a CNN is a hyper-parameter.

In both contexts, the words "kernel" and "filter" are roughly synonymous, so they are often used interchangeably. Furthermore, in both cases, the kernels are related to the convolution (or cross-correlation) operation. More specifically, the application of a filter, which is a function $h$, to an input, which is another function $f$, is equivalent to the convolution of $f$ and $h$. In mathematics, this is often denoted by $f \circledast h = g$, where $\circledast$ is the convolution operator and $g$ is the result of the convolution operation and is often called the "convolution" (of $f$ and $h$). In the case of image processing, $g$ is the filtered image. In the case of CNNs, $g$ is often called an activation map.

Have a look e.g. at Visualizing what ConvNets learn for some info regarding the visualisation of the kernels learned during training of a CNN.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks for your answer @nbro . You are saying "kernels are the learnable parameters of the model". My understanding was that we would also have a set of weights to back-prop, connecting the output of each layer to the input of the next one. Is this correct? $\endgroup$ – nvergontbij May 25 '19 at 17:16
  • 1
    $\begingroup$ @nvergontbij In the case of CNNs, this is not the case, but this would be true in the case of feed-forward neural networks (or multi-layer perceptrons). In the case of CNNs, the kernels (or filters) are the weights of the network. $\endgroup$ – nbro May 25 '19 at 20:14
1
$\begingroup$

So if your looking for filters with known effect, gaussian filters do smoothing, gabor filters are useful for edge detection, etc..

usually in deep learning models where things are trained from scratch, the filters are randomly initialized and then learned by the models training scheme. For the most part without using any of the well known kernels mentioned above

(clarification on for the most part): so filters arent initialized with the goal of knowing exactly which feature it will activate on, but what will assist the training procedure. Recently people have even trained ResNets w/o batchnorms by finding good initial points-- its an ongoing field of research.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.