0
$\begingroup$

In the paper, they first use the terms "up projection layer," and similarly for down projection, in this paragraph in the introduction:

Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks.

What does this refer to in terms of the architecture of a given LLM?

This paper focuses on the Falcon and OPT LLM models.

To try to understand this, I've read through the paper for other explanations, and I searched online. One suggestion I found is that there are special layers in a network that (speaking informally) increase or decrease the size of the resolution, but I don't think that would apply to these specific LLM models. (For example, a max-pooling layer may be seen as decreasing resolution.)

The image below is relevant, but I don't think it answers the question. Here is the paper in question.

Figure 7 from the paper

$\endgroup$

2 Answers 2

1
$\begingroup$

Up-project and down-project refer to the first and second feed forward layer respectively, found in each transformer block.

They use this in the context of sparsity in the feedforward layers in each transformer block.

Recall that each transformer block has two FF layers with ReLU activations. Each FF can be thought of as a "projection" to a different dimension. They noticed that with ReLU, the output of the first FF layer (the "intermediate values") tends to be very sparse:

The ReLU activation function naturally induces over 90% sparsity in the FFN’s intermediate outputs, which reduces the memory footprint for subsequent layers that utilize these sparse outputs. However, the preceding layer, namely the up project for OPT and Falcon, must be fully present in memory.

Because of this sparsity, you don't want to load the entire FFN weights, so the authors propose efficient ways to do this.

$\endgroup$
1
  • $\begingroup$ I suspect you are correct, and, in slightly more detail, that the authors want to load both the incoming weights for a particular neuron, as well as all the weights that will be multiplied by the output of that neuron in the next layer. One thing that made this hard for me to understand is that I don't think the terms "up project" or "down project" are standard for feedforward networks. At least not in the literature I have previously read. $\endgroup$
    – Tyler
    Jan 30 at 1:29
0
$\begingroup$

The answer could be found in the author's another paper. https://arxiv.org/pdf/2310.04564.pdf

For FFN(x) = linerar1(relu(linear0(x))), the first nn.Linear is the "up-project" , the second nn.Linear is the "down-project"

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .