Edge inference has a memory problem that does not get solved by better kernels. On a Jetson Nano, 4 GB of capacity decides which models run at all; on an 8 GB Orin Nano Super, bandwidth decides how fast they run. Every standard quantization format — GPTQ, AWQ, and llama.cpp’s own K- and I-quants — spends the same number of bits on every integer code of a matrix, even though those codes are wildly non-uniform in practice. The entropy is there. The formats just throw it away.
LazyWeights keeps the formats but changes the encoding. It tunes GPTQ so the resulting integer codes are compressible, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit sitting between DRAM and L2. GPU kernels run unchanged; DRAM now moves only coded bytes rather than full weights. The result is not a new quantization scheme but a transport and decode layer over an existing one.
The measured numbers are the argument. At 1% and 3.5% perplexity budgets, the coded models — Phi-3-mini, Mistral-7B and SmolLM-1.7B — are 5.7% to 31.4% smaller than the best llama.cpp format. A bit-exact 0.21 mm² ASIC decoder (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38%, rising to 54% and 44% when prompt-lookup speculative decoding is on top. On the Nano, LazyWeights needs 58–64% less time and 53–57% less energy per token than CPU llama.cpp, and streams Mistral-7B 6.6× faster at 3.89 bits per weight.
Two design details matter for anyone considering this. First, the decoder is placed between DRAM and L2, not in the GPU pipeline, so it does not touch kernel code — a rare hardware proposal that leaves the software stack intact. Second, the paper compares Huffman against rANS for the same decoder unit and finds Huffman better on rate and area at equal throughput, and the unit is bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102). Bit-exactness matters: the compression is lossless, so accuracy claims reduce to the perplexity budget of the underlying GPTQ tuning, not to a new approximation error.
The honest caveat is that this is a hardware-plus-firmware design, not a shipping product — it assumes an ASIC decoder can be placed in the memory path of a device you do not necessarily control. For anyone shipping an edge product where the DRAM bill dominates, though, the 5.7–31.4% model-size reduction and the 6.6× streaming speedup on a 4 GB part are hard numbers to ignore.
Paper: arXiv:2611.17108