# How are hidden layers counted / semantically defined?

I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer".

Using the following FFN as an example:

1. The weight matrices will be $$W_1$$ of size $$[4 \times 6]$$ and $$W_2$$ of size $$[6 \times 4]$$.
2. The bias vectors will be $$b_1$$ of size $$[6]$$ and $$b_2$$ of size $$[4]$$.

$$W_1 = \begin{pmatrix} 0.1 & 0.2 & 0.3 & 0.4 & 0.5 & 0.6 \\\ 0.7 & 0.8 & 0.9 & 1.0 & 1.1 & 1.2 \\\ 1.3 & 1.4 & 1.5 & 1.6 & 1.7 & 1.8 \\\ 1.9 & 2.0 & 2.1 & 2.2 & 2.3 & 2.4 \end{pmatrix}$$

$$b_1 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \\\ 0.05 \\\ 0.06 \end{pmatrix}$$

$$W_2 = \begin{pmatrix} 0.1 & 0.7 & 1.3 & 1.9 \\\ 0.2 & 0.8 & 1.4 & 2.0 \\\ 0.3 & 0.9 & 1.5 & 2.1 \\\ 0.4 & 1.0 & 1.6 & 2.2 \\\ 0.5 & 1.1 & 1.7 & 2.3 \\\ 0.6 & 1.2 & 1.8 & 2.4 \end{pmatrix}$$

$$b_2 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \end{pmatrix}$$

and just in case it matters, let's say that the input from layer normalization is:

$$\text{LN}(x_i) = \begin{pmatrix} -1.03927142 & 0.13949668 & 1.56694829 & -0.66717355 \\\ -0.97825977 & 0.09190721 & 1.59246789 & -0.70611533 \\\ -1.10306273 & 0.02854757 & 1.58729045 & -0.51277529 \end{pmatrix}$$

How many hidden layers are in the model? It's not clear to me if each row (function) in the weight matrix is a "layer" or if the whole process of multiplying $$LN(x_i)$$ by $$W_1$$ is a single layer?

As I understand it, $$W_2$$ is the output layer?

In a FFN, a layer is a set of functions $$f_1, f_2, \dots, f_n$$ like this $$f_j = \sigma(x_1 * w_{j1} + \dots + x_n * w_{jm})$$ (sometimes called neurons), where $$x_i$$ and $$w_i$$ are respectively the inputs and parameters, and $$\sigma$$ is the activation function. So, $$m$$ is the number of inputs each function receives, which is equal to the number of parameters, while $$n$$ is the number of functions in the layer. You might have noticed that I wrote $$w_{j1}$$ instead of $$w_1$$ - this is intentional - each function has its own parameters.

Now, can you write the computation of a layer, set of functions, as a matrix multiplication? Hint: yes

Last, notice that $$f_j$$ becomes $$x_j$$ for the successive layer. So you can stack layers. Neural networks are just composite parametrized functions.

• Let me know if this helps you address your specific question. Otherwise I can be more specific.
– nbro
Nov 10 at 2:26

For your example, $$W_2$$, $$b_2$$, and whatever activation function you use would collectively be considered the last layer. But for a more complicated architecture, such as a stack of LSTMs, the set of (matrix-vector multiplications + activations) that are necessary for a single LSTM gate might be considered a "layer".

There's no formal definition of "layer". We call a group of calculations a "layer" whenever thinking about them as a single unit make it easier to reason about how the network as a whole works.