06 / THE HARDWARE PUZZLE
Model memory
Pick a model, a precision, and a memory budget. Then make room for the cache and see where the space goes.
Try a challenge. Or change whatever you want.
0 / 4 SOLVEDChoose a build
Try a bigger model, more cached tokens, or fewer bits. See what each part takes up.
8B / 16-BIT / 8 GiB
doesn’t fit in this crate
The crate updates as you go. Check the numbers when you’re ready.
THE SAME BUILD / FOUR WAYS
Compare weight precision
Same model, cache and reserve.
Only weight precision changes.
Two ideas, side by side.
The math, the assumptions, and what this does not tell you
Where the memory goes.
Ideal weight storage: parameters × bits ÷ 8 ÷ 2³⁰ GiB. A billion parameters means 10⁹ parameters; a GiB means 2³⁰ bytes.
Illustrative KV cache: 2 × layers × KV heads × head dimension × cached tokens × sequences × bytes per value. This is a conventional full-attention key/value cache calculation.
Manual reserve: a chosen allowance that is added to the total. It is not calculated from a runtime.
The assumptions in this example.
The cache always assumes 32 layers, 8 KV heads, 128 dimensions per head and 16-bit cache values. The parameter count changes only the weights. These independent knobs do not describe a named or benchmarked model.
The estimate omits allocator behavior, activations, intermediate buffers, quantization metadata, architecture-specific caches, and device memory already in use. Real systems may partition or offload storage. A green crate is a solved arithmetic budget, not a guarantee that inference will run.
More bits does not automatically mean a better answer. The puzzle’s “highest precision” rule is a teaching objective, not a universal deployment recommendation.
Read the cache shape reference ↗Solved briefs and your last workshop build stay in this browser.