How are hidden layers counted / semantically defined?

Question

I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer".

Using the following FFN as an example:

1. The weight matrices will be $W_1$ of size $[4 \times 6]$ and $W_2$ of size $[6 \times 4]$.
2. The bias vectors will be $b_1$ of size $[6]$ and $b_2$ of size $[4]$.

$$W_1 = \begin{pmatrix} 0.1 & 0.2 & 0.3 & 0.4 & 0.5 & 0.6 \\\ 0.7 & 0.8 & 0.9 & 1.0 & 1.1 & 1.2 \\\ 1.3 & 1.4 & 1.5 & 1.6 & 1.7 & 1.8 \\\ 1.9 & 2.0 & 2.1 & 2.2 & 2.3 & 2.4 \end{pmatrix}$$

$$b_1 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \\\ 0.05 \\\ 0.06 \end{pmatrix}$$

$$W_2 = \begin{pmatrix} 0.1 & 0.7 & 1.3 & 1.9 \\\ 0.2 & 0.8 & 1.4 & 2.0 \\\ 0.3 & 0.9 & 1.5 & 2.1 \\\ 0.4 & 1.0 & 1.6 & 2.2 \\\ 0.5 & 1.1 & 1.7 & 2.3 \\\ 0.6 & 1.2 & 1.8 & 2.4 \end{pmatrix}$$

$$b_2 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \end{pmatrix}$$

and just in case it matters, let's say that the input from layer normalization is:

$$ \text{LN}(x_i) = \begin{pmatrix} -1.03927142 & 0.13949668 & 1.56694829 & -0.66717355 \\\ -0.97825977 & 0.09190721 & 1.59246789 & -0.70611533 \\\ -1.10306273 & 0.02854757 & 1.58729045 & -0.51277529 \end{pmatrix} $$

How many hidden layers are in the model? It's not clear to me if each row (function) in the weight matrix is a "layer" or if the whole process of multiplying $LN(x_i)$ by $W_1$ is a single layer?

As I understand it, $W_2$ is the output layer?

Answer 1

In a FFN, a layer is a set of functions $f_1, f_2, \dots, f_n$ like this $f_j = \sigma(x_1 * w_{j1} + \dots + x_n * w_{jm})$ (sometimes called neurons), where $x_i$ and $w_i$ are respectively the inputs and parameters, and $\sigma$ is the activation function. So, $m$ is the number of inputs each function receives, which is equal to the number of parameters, while $n$ is the number of functions in the layer. You might have noticed that I wrote $w_{j1}$ instead of $w_1$ - this is intentional - each function has its own parameters.

Now, can you write the computation of a layer, set of functions, as a matrix multiplication? Hint: yes

Last, notice that $f_j$ becomes $x_j$ for the successive layer. So you can stack layers. Neural networks are just composite parametrized functions.

Answer 2

For your example, $W_2$, $b_2$, and whatever activation function you use would collectively be considered the last layer. But for a more complicated architecture, such as a stack of LSTMs, the set of (matrix-vector multiplications + activations) that are necessary for a single LSTM gate might be considered a "layer".

There's no formal definition of "layer". We call a group of calculations a "layer" whenever thinking about them as a single unit make it easier to reason about how the network as a whole works.