Memory bandwidth, mixture-of-experts, and why the obvious choice was the wrong one.
2026-08-14
I spent an evening working out which open model to run locally on a 48 GB M4 Pro, got the answer confidently wrong, and then got it right. The wrong turn is the useful part.
Generation speed on Apple Silicon is bound by memory bandwidth, not compute. Roughly, tokens per second is bandwidth divided by the bytes read per token. An M4 Pro has about 273 GB/s. A dense model reads all of itself for every token; a mixture-of-experts model reads only its active parameters.
That difference is enormous. A 30B dense model at 4-bit is about 15 GB per token, so roughly 18 tokens per second at the theoretical ceiling. A model with 3B active parameters reads about 1.5 GB and could in principle manage ten times that. So the obvious move is a big MoE with a small active count, and that is exactly what I picked.
I reasoned entirely from architecture and never checked whether the model was any good. When I finally looked at the benchmark index, the coding-specific MoE I had chosen scored 14. A 27B dense model in the same size class scored 52. Nearly four times higher, and within touching distance of models I cannot run at all.
The dense model is genuinely about five times slower. I took it anyway, because for agentic work a wrong answer costs a whole loop and slow-but-right beats fast-but-wrong on wall clock. But I would have shipped the fast one on architectural elegance alone if I had not gone and looked.
Size on disk is not the parameter count. The 4-bit build of the 80B model I first wanted is 44.9 GB. On a 48 GB machine that leaves nothing for the OS or the KV cache, which grows with context and matters enormously for long agentic runs. Check the actual repository size rather than multiplying parameters by a half.
And published tokens-per-second figures are measured on datacentre hardware serving an API. They tell you nothing about your laptop. Work it out from your own memory bandwidth instead.