The problem the paper picks

A 7B parameter model in half precision is about 14 GB of weights, and a phone or a base laptop does not have that much memory to spare. The usual answer is to shrink the model, either by quantising it or by training a smaller one. Keivan Alizadeh, Mehrdad Farajtabar and six colleagues at Apple take a different route. They leave the weights at full size, keep them on flash storage, and load only what a given token needs into DRAM.

The reason this is hard is the shape of flash bandwidth. On an M1 Max the paper measures more than 6 GiB/s for a large sequential read, but random reads of small chunks are dramatically slower. A naive design that fetches individual weight rows as they are needed spends nearly all of its time waiting on we/O. The paper builds an explicit cost model around this and then optimises two quantities, how many bytes are read per token and how large each read is.

Sparsity does most of the work

The first lever is that feed-forward layers in ReLU-style models are mostly idle. The authors report 97 percent sparsity in the FFN layers of OPT 6.7B and 95 percent in a relufied Falcon 7B. A Llama 2 7B variant sparsified with FATReLU lands at 90 percent. If you know in advance which neurons will fire, you only need to load those rows, and the attention weights, which are a smaller share of the model, can stay resident in DRAM.

Knowing in advance is the trick. The paper trains a low rank predictor per layer, with ranks between 128 and 1152 depending on how sensitive the layer is, that guesses which FFN neurons will be active from the current hidden state. The predictor overshoots on purpose, predicting about three times the true sparsity so that it rarely misses a neuron that matters, and the authors report near zero accuracy degradation on their benchmarks. The result is that each token loads roughly 2.4 percent of FFN neurons for OPT 6.7B and 3.1 percent for Falcon 7B.

Windowing and bundling

Loading 2.4 percent of the FFN per token is still a lot of small reads if you throw the cache away every step. The second lever, which the paper calls windowing, keeps a DRAM cache of the weight rows predicted for the last k tokens, with k of 4 or 5 in the experiments. Because consecutive tokens activate overlapping sets of neurons, most of what the next token needs is already in memory and only the difference gets loaded. Neurons that fall out of the window are evicted in linear time by overwriting their slots with the most recent entries, which avoids reallocating memory on every step.

The third lever is about read size. In a transformer FFN the up projection column for a neuron and the down projection row for the same neuron are always needed together, so the authors store them adjacently in flash and fetch them as one chunk. This row-column bundling doubles the size of each read to around 32 to 35 KiB, which sits in the region where flash throughput stops being dominated by per-read overhead. None of this changes the maths the model does. It only changes what the storage controller sees.

What the numbers look like

The experiments assume about half of the available DRAM or GPU memory is set aside for the model, so a device can run a model up to twice the size of what fits. On an M1 Max CPU, generating one token of OPT 6.7B naively takes 3,182 ms, of which 2,196 ms is we/O. With all three techniques the total drops to 669 ms with 105 ms of we/O. On an RTX 4090 the naive figure is 2,218 ms, again dominated by the same 2,196 ms of loading, and the optimised version is 84 ms with 30 ms of we/O. Adding speculative decoding brings the GPU number to 60 ms.

The abstract summarises this as a 4 to 5 times speedup on CPU and 20 to 25 times on GPU relative to naive loading. We would read those multipliers with the baseline in mind. The comparison is against a deliberately simple approach that fetches from flash on every token, and the honest question is how far the optimised version sits from a model that fits entirely in memory. The tables answer that too, and the gap is real but no longer absurd. Around 670 ms per token on a laptop CPU for a 7B model with half its weights on disk is usable for short interactions.

Why this reads as a strategy document

Apple does not usually publish inference papers about other people's models, and this one benchmarks OPT, Falcon, Persimmon, Phi-2 and Llama 2 on Apple hardware with Apple SSDs. Every design choice is about a device with a fixed, small memory budget and a fast local flash chip. That describes an iPhone and a base model MacBook far more than it describes a datacentre. The paper says nothing about products, but the constraints it chooses are the constraints of the devices the company sells.

The dependency we would watch is the sparsity assumption. The whole scheme leans on ReLU-style FFN layers where most neurons are quiet, and current open models mostly use gated activations such as SwiGLU that do not give you 95 percent sparsity for free. The Llama 2 result required FATReLU sparsification first, and the paper does not spend much time on what that costs in quality. If someone reproduces the accuracy of a stock Llama 2 7B after relufication at the 90 percent sparsity the method needs, this becomes a general recipe rather than a demonstration on models chosen to suit it. That reproduction is the experiment we would want to see next.

Sources

  1. arXiv: LLM in a flash: Efficient Large Language Model Inference with Limited Memory (Alizadeh et al.)