We generally encounter the following statement several times

The input vector is first fed into a fully connected layer......

Since linear activation functions, such as identity function, can so considered as an activation functions, a fully connected layer can be considered just as an Affine transformation if the fully connected layer uses linear activation function.

So, in theory, a fully connected layer can refer to the following

  1. Just an affine transformation
  2. Affine transformation followed by a nonlinear activation function

Do authors generally choose to use "fully connected layer" for case 2 only or for both cases 1 and 2?

  • 1
    $\begingroup$ Affine transformations have this property that multiple affine transformations are equivalent to one affine transformation, so you gain nothing by stacking such layers. $\endgroup$
    – user253751
    Sep 16 at 8:06
  • $\begingroup$ @user253751 consider a single one. $\endgroup$
    – hanugm
    Sep 16 at 8:23
  • 1
    $\begingroup$ According to my experience, it refers to the 2nd case. When you're talking of neural networks, it's implicitly assumed you use a non-linear activation function at the end of each layer. $\endgroup$
    – SpiderRico
    Sep 16 at 20:22

Yes, typically, a fully connected layer is an affine transformation, which can or not be followed by a non-linear activation function, but, in many (if not most) cases, it's followed by a non-linearity, such as ReLU, sigmoid, or tanh (an exception is when you do regression), which is what makes the neural network be able to approximate non-linear/complicated functions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.