0
$\begingroup$

Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most models can't run inference on consumer hardware. Another problem is that they do not have an explicit world model, so that you can not be 100% sure if a response is factual or hallucinated.

Along the old neat-scruffy axis, I wonder if there are alternative architectures that are a bit more in the scruffy direction. For example, I could imagine that you use deep learning only for "language understanding", but for "world knowledge" you would have a fact database like a triplestore. The language model of such a system would be much smaller, only needing to reproduce basic grammar, and the facts would be explicit instead of implicit.

So are there any - potentially competitive - NLP models that are not of the autoregressive transformer type, scruffier, smaller, and yet operating in a similar space to GPT-3, LLaMA and co.?

$\endgroup$

2 Answers 2

0
$\begingroup$

External memory indexed from attention - https://arxiv.org/pdf/2112.04426.pdf

Regarding minimizing cost of autoregression, one could do encoder-decoder (eg T5) architecture, but with a twist: Make the decoder deliberately much smaller network than the encoder which is doing most of the heavy lifting, but running only once. Such asymmetry works on assumption that "speaking" a concept from the latent space vector is a simpler task compared to "understanding" it done in the encoder.

If this paper is correct, this allows you to run T5-FLAN at 6x inference speed at a cost of marginal (<5%) loss. The paper utilizes pruning (ie downsizing pre-trained model), better scaling might be possible if trained like this from scratch.

$\endgroup$
0
$\begingroup$

It sounds like you're talking about KNN-LMs and Retrieval Augmented Generation.

KNN-LMs

These are models that, during inference time, look up the embedding representation of their context in a datastore. They use the retrieved next-token-prediction in addition to their regular output to make the final prediction.

KNN-LM diagram from https://arxiv.org/pdf/1911.00172.pdf The above figure is from the KNN-LM paper.

Retrieval Augmented Models

This is an extremely general class of models, which basically involve the retrieval of some information which gets added to the context before the model proceeds with regular autoregressive generation.

This retrieved information can be web pages (e.g., Bing Chat), documents from an existing database (see: open-domain question answering), or information from a knowledge graph. This paper does knowledge graph retrieval, then adds an encoded subgraph into the context.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .