When I start to learn about RAG, I can't understand the difference between the RAG-Sequence Model and RAG-Token Model.

First of all, I see that for RAG-Sequence Model we use just one document to generate the hole output and for RAG-Token Model for each token we use all the k document chosen.

But, when I see the equation in the paper, I can't understand why they make a summation in the case of RAG-Sequence Model while we generate the output from one document.

enter image description here


1 Answer 1


In the first equation, you can see that in the sequence likelihood (the big product) $z$ is constant, which means that you first find which are the interesting documents, than you give your LLM these document, and you ask it to answer (for the whole generation, these documents are always the same)

In the second case, you can see that for every step ($i$), you have a corresponding $z_i$ (a set of documents) that is used in the summation. However, you can see that that summation is just an expectation for the $y_i$ token. This means that, if earlier you first find K documents and then generate an answer, now you have to find interesting documents every time you have to sample a token.

In other words:

  • in the first case: (1) sample K documents based on query (2) ask the LLM to generate the first token (3) concat the new token to the generated sentence (4) repeat 2 until EOS token is chosen
  • in the second case: (1) find top K documents based on the current generation&query (2) feed the documents to the LLM, (3) generate next token (4) concat the new token to the current generated sequence, (5) repeat 1 until EOS is sampled

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .