QLoRA: fine-tuning a 65B model on one GPU

The number that matters

Fine-tuning a 65 billion parameter model used to need more than 780GB of GPU memory in 16-bit, which is a rack, not a workstation. The QLoRA paper, posted to arXiv on May 23 by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, brings that to a single 48GB GPU with no measured loss in task performance relative to 16-bit fine-tuning. Their Guanaco models, trained this way on LLaMA, reach 99.3 percent of ChatGPT's score on the Vicuna benchmark after 24 hours on one GPU.

We want to be careful with that last figure, and so are the authors. The Vicuna benchmark uses GPT-4 as judge over a small prompt set, and the paper spends a section arguing that current chatbot benchmarks are not trustworthy for measuring absolute performance. Take the 99.3 percent as evidence that 4-bit fine-tuning does not wreck a model, not as evidence that a hobbyist has matched OpenAI.

Where the memory goes

The baseline idea is LoRA, from a 2021 paper: freeze the pretrained weights and train small low-rank adapter matrices alongside them. That already removes the need to store optimizer state for the full model. But the frozen weights still have to sit in memory, and in 16-bit a 65B model is 130GB before you have done anything. So the frozen base has to shrink.

QLoRA stores the base model in 4 bits and dequantizes on the fly to 16-bit for each forward and backward pass. Gradients flow through the dequantized weights into the adapters, which stay in 16-bit. The base never changes, so quantizing it once up front costs nothing during training. Everything else in the paper is about making that 4-bit storage as faithful and as small as possible.

NormalFloat, double quantization, paged optimizers

The first piece is a data type. Pretrained weights are roughly normally distributed, and a uniform 4-bit integer grid wastes levels on the tails where few weights live. NF4, 4-bit NormalFloat, places its 16 levels at the quantiles of a normal distribution, so each level covers an equal share of the weights. The authors describe it as information theoretically optimal for normally distributed values, and in their ablations it beats plain 4-bit float and integer formats.

The second piece is bookkeeping. Blockwise quantization stores a scaling constant per block of weights, and with small blocks those constants add up. Double quantization quantizes the constants themselves, storing them in 8-bit with a second, coarser set of constants above them. It sounds like a rounding error but on a 65B model it is the difference between fitting and not fitting.

The third piece handles spikes rather than steady state. Memory use during training is not flat, and a long sequence can briefly push the optimizer state past the card's capacity and crash the run. Paged optimizers use NVIDIA unified memory to page optimizer state to CPU RAM when the GPU fills and page it back when needed. It costs speed on the pages that move and buys the ability to train through a spike instead of dying on it.

What they did with a thousand cheap runs

Once a fine-tune costs a day on one card, you can afford to do a lot of them, and the paper's second half is the result. The authors report fine-tuning over 1,000 models across several instruction datasets, model sizes and architectures. The finding we keep coming back to is that a high-quality instruction set produced state-of-the-art results even with a smaller model, which the authors put more weight on than dataset size.

They also used the runs to test the evaluation itself, and their conclusion is that current chatbot benchmarks are not trustworthy for measuring performance levels. That section is worth reading on its own if you are building an evaluation pipeline right now, because a lot of people are about to trust a model as a judge without checking.

Why this changes who can do the work

Before this month, fine-tuning a model in the 30 to 65 billion range was an activity for a lab with a multi-node cluster and someone to babysit it. LoRA had already made 7B and 13B accessible on a single consumer card. QLoRA moves the ceiling up by a factor of five or so without moving the hardware requirement. A researcher with one A6000 or a rented A100 can now try an idea on a 65B model overnight and see the result in the morning.

The code, the CUDA kernels for 4-bit training and the Guanaco weights are all released. That matters for reproducibility as much as for convenience. Anyone can check the 99.3 percent claim on the same benchmark, and anyone can check the NF4 ablation on their own model.

What we would want next is a study of what 4-bit base weights do to fine-tuning on tasks that need precise numerical reasoning, where quantization error might matter more than it does for chat. The paper's evaluations are mostly on instruction following and academic benchmarks like MMLU, and a 4-bit model that loses nothing on MMLU could still lose something on arithmetic. That is a cheap experiment now. Which is the point.