0
$\begingroup$

We generally encounter the following statement several times

The input vector is first fed into a fully connected layer......

Since linear activation functions, such as identity function, can so considered as an activation functions, a fully connected layer can be considered just as an Affine transformation if the fully connected layer uses linear activation function.

So, in theory, a fully connected layer can refer to the following

  1. Just an affine transformation
  2. Affine transformation followed by a nonlinear activation function

Do authors generally choose to use "fully connected layer" for case 2 only or for both cases 1 and 2?

$\endgroup$
3
  • 1
    $\begingroup$ Affine transformations have this property that multiple affine transformations are equivalent to one affine transformation, so you gain nothing by stacking such layers. $\endgroup$ Sep 16, 2021 at 8:06
  • $\begingroup$ @user253751 consider a single one. $\endgroup$
    – hanugm
    Sep 16, 2021 at 8:23
  • 1
    $\begingroup$ According to my experience, it refers to the 2nd case. When you're talking of neural networks, it's implicitly assumed you use a non-linear activation function at the end of each layer. $\endgroup$
    – SpiderRico
    Sep 16, 2021 at 20:22

1 Answer 1

1
$\begingroup$

Yes, typically, a fully connected layer is an affine transformation, which can or not be followed by a non-linear activation function, but, in many (if not most) cases, it's followed by a non-linearity, such as ReLU, sigmoid, or tanh (an exception is when you do regression), which is what makes the neural network be able to approximate non-linear/complicated functions.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .