B.BRANDON FRANK HAN-CHWEEN HUANG← ProjectsMy account

06 / THE HARDWARE PUZZLE

Model memory

Pick a model, a precision, and a memory budget. Then make room for the cache and see where the space goes.

Try a challenge. Or change whatever you want.

0 / 4 SOLVED
FREE PLAY / NO WRONG TURNS

Choose a build

Try a bigger model, more cached tokens, or fewer bits. See what each part takes up.

THE MEMORY CRATE

8B / 16-BIT / 8 GiB

OVER CAPACITY
14.90GiB of weights
TOTAL PLANNED—of 8 GiB
64 CELLS / EXACT FRACTIONS
Weights—
KV cache—
Manual reserve—
Unallocated—

The crate updates as you go. Check the numbers when you’re ready.

THE SAME BUILD / FOUR WAYS

Compare weight precision

Same model, cache and reserve.
Only weight precision changes.

The math, the assumptions, and what this does not tell you

Where the memory goes.

Ideal weight storage: parameters × bits ÷ 8 ÷ 2³⁰ GiB. A billion parameters means 10⁹ parameters; a GiB means 2³⁰ bytes.

Illustrative KV cache: 2 × layers × KV heads × head dimension × cached tokens × sequences × bytes per value. This is a conventional full-attention key/value cache calculation.

Manual reserve: a chosen allowance that is added to the total. It is not calculated from a runtime.

The assumptions in this example.

The cache always assumes 32 layers, 8 KV heads, 128 dimensions per head and 16-bit cache values. The parameter count changes only the weights. These independent knobs do not describe a named or benchmarked model.

The estimate omits allocator behavior, activations, intermediate buffers, quantization metadata, architecture-specific caches, and device memory already in use. Real systems may partition or offload storage. A green crate is a solved arithmetic budget, not a guarantee that inference will run.

More bits does not automatically mean a better answer. The puzzle’s “highest precision” rule is a teaching objective, not a universal deployment recommendation.

Read the cache shape reference ↗

Solved briefs and your last workshop build stay in this browser.