Would self-hosting ChatGPT be feasible, w.r.t. computation costs?

Question

Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?

Assumptions

1. I assume getting output based on some input is, at least, hundreds of times faster than training such a model.
2. I assume no additional output parsing/input limitations are used. In particular I can imagine all the boiler plate to keep the ChatGPT model(s) acting politically correct etc. may be a significant overhead. This is to be ignored for this question.

Data

So far I've found the ChatGPT 3.5 model to have 175 billion parameters:

Though I do not yet know how large that is in Mb nor do I have an idea on how long generating an output would typically take.

Answer 1

Here, I assume the ChatGPT architecture is similar to the published GPT-3 model [1].

The Transformer architecture used for GPT3 is much more memory-bound than compute-bound at a high parameter count and a low context size. A large batch size (3.2M across ~few thousands GPUs for GPT-3) is used during training, but relatively low computational power is needed at the inference time for non-batched applications such as real-time chat. OpenAI is probably batching multiple users' requests to improve the throughput.

GPT-3 has 175B parameters and would require 326GiB memory to store the weight in float16. Quantization methods are proposed to reduce the memory required to store model weights, where the model weights are stored in lower precision. In the GPTQ paper [2], the authors have shown that large GPT models perform almost as good as the original model with 4-bit per parameter, reducing the 175B model to less than 90GiB. 3-bit quantization is also considered, reducing the size to less than 70GiB. Note that additional memory required for internal states is small compared to the weights, especially if a memory-efficient attention is used.

Recently, such quantization methods have gained attention due to the release of the pre-trained LLaMA models. Some people are running 65B LLaMA model even CPU-only [3] [4] with a reasonable throughput.

Therefore, for the question of whether it would be possible to run a 175B model in a home computer, I would say it is currently stretch but certainly possible if somewhat slow responses are acceptable.

• [1]: Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901. https://arxiv.org/abs/2005.14165
• [2]: Frantar, Elias, et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." (2022). https://arxiv.org/abs/2210.17323
• [3]: Satyajit Ghana on Twitter
• [4]: Taiga Takano on Twitter

Answer 2

Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?

No, assuming it's regular, consumer hardware. However, some open-source alternatives aim to be runnable on consumer hardware, e.g. https://github.com/LAION-AI/Open-Assistant (not released yet):

We want to do this in a way that is open and accessible, which means we must not only build a great assistant, but also make it small and efficient enough to run on consumer hardware.

There exist however smaller language models, e.g. see this survey of small language models.