It is easy to see the amount of disk space consumed by an LLM model (downloaded from huggingface, for instance). Just go in the relevant directory and check the file sizes.
How can I estimate the amount of GPU RAM required to run the model?
For example, if the Falcon 7B model takes around 14GB of storage, how much GPU RAM should suffice for it?
It varies depending on various factors such as quantization. My rough rule of thumb is memory need is 2-4x of the disk size. Just as an example, the model at https://huggingface.co/TheBloke/wizardLM-7B-HF/tree/main is about 14GB on disk and it used ~30GB (on CPU) just to load. I am sure inference will increase this further.
It really depends on what you want to do with it, how you load it and what else you add.
For inference at half-precision (16 bit per param) a 7B param models should be 7 billion * 2 bytes per param or 14GB. If you load it at full precision it will be a lot more. For training or tuning there are a lot of other states added for the optimizer and gradients. For a detailed discussion of what all is required for tuning and how to use approaches to train at scale please see the paper "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (https://arxiv.org/abs/1910.02054). Deepspeed is an implementation of ZeRO.
Additionally, for training you can use "LoRA: Low-Rank Adaptation of Large Language Models" (https://arxiv.org/abs/1910.02054) to reduce the amount of trainable parameters. As an example, on a 7B parameter model I get a reduction of 6.5 billion param values to about 540,000. PEFT is an implementation of LoRA.
As a reference, when I execute:
import torch
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("decapoda-research/llama-7b-hf", torch_dtype=torch.bfloat16)
model.cuda()
And I used nvidia-smi to check memory, my A100 is using 13845MiB.
NOTE: If I do not use the torch_dtype=torch.bfloat16 then the GPU utilization is 26729MiB. Most folks inference at 16 bit or even 8 bit quantization. For example: https://huggingface.co/blog/hf-bitsandbytes-integration/
The following formula can be used to calculate GPU memory for serving LLMs:
Source and more examples: https://www.substratus.ai/blog/calculating-gpu-memory-for-llm
Disclaimer: I'm the author of that blog
