4
$\begingroup$

I know they are not the same in working, but an input layer sends the input to x neurons with a set of weights, based of these weights and the activation layer, it produces an output that can be fed to the next layer.

Are filters not the same in the way that they convert an "image" to a new "image" based on the weights that are in that filter? And that the next layers uses these new "images"?

$\endgroup$
3
$\begingroup$

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer

Differences

The neurons of these two types of layers have two key differences. These are that the convolution layers implement:

  • Sparse connectivity, i.e. each neuron is connected only to an area of the input, not the whole.
  • Weight sharing, i.e. similar connections end up having the same weights. This is usually visualized as the same filter traversing the image.

Besides these two key differences, there are some other technical details, e.g. how the biases are implemented. Other than that they perform the same operation.

What causes some confusion is that the input of a CNN is usually 2 or 3-dimensional, while a FC is usually 1-dimensional. These aren't mandatory however. To better help visualize the differences between the two I made a couple of figures illustrating the differences between a conv-layer and a FC one, both in 1D.

Sparse connectivity

On the left are two FC neural networks, on the right are layers with sparse connections.

Weight sharing

On the left is a sparsely connected network. The colors represent the different values of the weights. On the right is the same network with weight-sharing. Note that similar weights (i.e. arrows with the same direction in each layer) have the same value.


To answer your other questions:

Are filters not the same in the way that they convert an "image" to a new "image" based on the weights that are in that filter? And that the next layers uses these new "images"?

Yes, if the input of a convolution layer is an image, so will the output. The next layer will also operate on an image.

However, I'd like to note that not all convolution layers accept images as their inputs. There are 1D and 3D conv layers as well.

| improve this answer | |
$\endgroup$
  • $\begingroup$ This is a good answer, but I think what confuses more people is the fact that nobody explicitly says what a neuron is in a CNN. Only the weights (i.e. the filters) are usually mentioned. Although you try to explain this with a simplified diagram, I think it would be really useful to explicitly state what the neurons in a typical CNN are. Are they the input (i.e. feature maps)? Are they the filters? Or are they just implicitly represented? Or what? How are these implicitly represented neurons specifically related to the filters? I haven't yet seen a good article that precisely describes this. $\endgroup$ – nbro Mar 25 at 13:41
  • $\begingroup$ To be more precise, in an FFNN (or MLP), the neurons are easily identified, i.e. they are the "imaginary units" that compute the linear combination of the inputs followed by a non-linear activation function. However, in a CNN, what are the neurons? Are the "imaginary units" that perform a dot product between a filter and a single part of the feature map? So, is the output of a neuron in a CNN a number? Or maybe a neuron is represented by the application of the convolution with a filter and all parts of the feature map (i.e. input) and, in that case, would the output of a neuron be a vector? $\endgroup$ – nbro Mar 25 at 13:47
  • $\begingroup$ You are correct that the terminology neuron, isn't broadly used in CNNs. However, If we want to use the equivalent to a neuron in FCNNs, the neuron is essentially the kernel, i.e. the function that computes the weighted average of its inputs. The two differences between convolutional neurons and FC neurons are that the former don't have weights for all their inputs, rather only a region of the input and that the former share their weights between neurons. This can be alternatively thought of as being the same neuron (i.e. the same kernel that passes through the input). $\endgroup$ – Djib2011 Mar 25 at 19:56
  • $\begingroup$ I edited the question to describe this more clearly. $\endgroup$ – Djib2011 Mar 25 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.