0
$\begingroup$

Why Fully-Connected Neural Network is not always better than Convolutional Neural Network?

FCNN is easily overfitting due to many params, then why didn't it reduce the params to reduce overfitting.

If eventually CNN will be flattened into FCNN, that's mean 2D layers (Conv2D, MaxPool, etc) of CNN is basically just 1D layers vector.

Also CNN is not fully connected when doing convolution, except the kernel size is 1×1 with stride 1 which basically same as fully connected layer. We know that fully connected means better.

And I think kernel size 1×1 and stride 1 in CNN (which basically equal same as FCNN) is performing better even though overfitting.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Two posts of mine on Cross Validated seem related: stats.stackexchange.com/questions/271198/… stats.stackexchange.com/questions/559113/… $//$ Did you mean to included the fully-convolutional-networks (FCN) tag? FCN and FCNN are not the same, and I think you mean FCNN. $\endgroup$
    – Dave
    Commented Feb 17, 2023 at 1:52
  • $\begingroup$ Do you know about the "no free lunch theorem"? Considering it, I think the question makes no sense, there is no better between layers, it all depends on the data and task. $\endgroup$
    – Dr. Snoopy
    Commented Feb 17, 2023 at 10:40
  • $\begingroup$ @Dr.Snoopy assume using same data and same task. $\endgroup$
    – Muhammad Ikhwan Perwira
    Commented Feb 17, 2023 at 23:12
  • $\begingroup$ That assumption makes no sense, as these layers do not do the same task or receive the same input data. $\endgroup$
    – Dr. Snoopy
    Commented Feb 17, 2023 at 23:27
  • $\begingroup$ @Dr.Snoopy it's just preprocessing data issue, you can transform 1D data into 2D (image) so it compatible with CNN input and vice versa for FCNN $\endgroup$
    – Muhammad Ikhwan Perwira
    Commented Feb 18, 2023 at 3:50

1 Answer 1

Reset to default
2
$\begingroup$

Why Fully-Connected Neural Network is not always better than Convolutional Neural Network?

The main reason why in many cases, a CNN will outperform a fully-connected (FC) neural network, i.e. MLP, is grounded in symmetry. For example, a CNN responds very naturally to image translations. This behavior is called translational equivariance which arises from shifting the same weights (i.e. the convolutional kernel) over the space of an image. The result is a model that - when trained on images where an object is in the center - will generalize to images where an object is off-center. This happens even though such images are not in your training set. An MLP cannot do this, it would have be trained on a dataset containing the objects in all possible positions in order to generalize nicely. As a side note: Some architectures like the transformer use FC layers in the style of a convolution as well, because the exact same weights are applied to all input tokens individually. So a model doesn't have to be your typical 2D-ConvNet to be a type of CNN.

FCNN is easily overfitting due to many params, then why didn't it reduce the params to reduce overfitting.

The model does not 'know' that it is overfitting because all it ever sees in training is the training dataset and an overfitted model performs great on this data. Another way to look at this is to view backpropagation as finding the path of least resistance to the optimal model. In many cases (especially with large models for small datasets) it is easier for the model to memorize the training samples instead of finding a very general solution.

We know that fully connected means better.

All layers have benefits and drawbacks, you can't really call any layer generally better, a fully-connected layer can do some things better than a CNN and other things (e.g. images) not so much.

And I think kernel size 1×1 and stride 1 in CNN (which basically equal same as FCNN) is performing better even though overfitting.

An overfitted model might look nice in training, but as soon as you show the model novel samples, these are much more likely going to be misclassified. Especially messy real-world data will likely result in bad model performance when you actually want to use such a model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ An MLP can learn just the same functions that CNN does. After all the CNN filters are just neurons aranged together. The big difference is that the weights are shared between the neurons. So you can just learn the same function with an MLP, but the neurons somehow (part of them) must learn the same weights. $\endgroup$
    – Antonios Sarikas
    Commented May 27, 2023 at 19:35
  • $\begingroup$ Exactly, this is where the symmetry idea manifests: If you do not want to include all possible positions of objects in images in your trainingset, you choose a local operator that naturally models the underlying symmetry. In this case translational symmetry. $\endgroup$
    – Chillston
    Commented May 29, 2023 at 9:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .