We Taught a Dead Server to Run a Frontier Model

There’s a server in my basement that has no business running a modern language model. It’s a repurposed HP StoreVirtual storage box, roughly thirteen years old, two Ivy Bridge Xeons, no GPU. It was built to hold disks, not do math. As of this week it runs Google’s Gemma 4, a 26-billion-parameter model, at about five tokens per second. Reading speed.

Anybody can rent a GPU. It’s harder to take a frontier model and a dead enterprise box and make them meet in the middle. But we’re nerds, and finding solutions that work with what we’ve already got is a point of pride around here.

The post that started it

A couple of weeks ago a piece called “A 10 year old Xeon is all you need” made the rounds on Hacker News. The author runs Gemma 4 on a single 2016 Xeon with no GPU and 128 GB of slow DDR3, using ik_llama.cpp and about 25 carefully chosen flags. It’s a great read, and it leans on every trick in the modern inference playbook: speculative decoding, CPU-aware mixture-of-experts routing, flash attention ported to the CPU, run-time weight repacking. Real engineering.

“I have a Xeon too,” I thought. Several, in fact. So I tried it. It didn’t run.

What an AI agent is actually good for

The build died on startup. I handed the failure to Claude and asked what was wrong. The answer came back fast and specific. The author’s 2016 chip is a Broadwell part. Mine are Ivy Bridge, the generation Intel calls “v2.” The fast kernels in that fork assume AVX2 and FMA3, instruction sets that didn’t ship until Haswell, the “v3” generation, in 2014. My CPUs are older than the instructions the code was written against. The optimized paths weren’t there to execute.

So I asked the obvious follow-up: can we make it run anyway? I’d already taken a first swing with a free model that got close but couldn’t land it. Claude picked up that half-finished approach, agreed it was the right one, and finished it off, reworking the hot paths so they fall back cleanly on a pre-AVX2 chip instead of reaching for instructions that aren’t there.

This is the part I care about. Nobody typed “write me a prompt” and got a working patch back. Somebody had to read another person’s performance-critical C++, work out why a kernel wasn’t valid on this particular microarchitecture, and route around it without throwing away the optimizations that made the fork worth using. Claude did that work. My job was narrower: run the right experiments and recognize when the output was finally correct. I came away impressed.

The result

Gemma 4’s 26B mixture-of-experts model now generates text at reading speed on hardware that was retired before the model’s architecture existed. The original write-up never published a tokens-per-second figure, just “reading speed,” so here’s the concrete one: about five tokens a second on thirteen-year-old silicon, for borderline free.

Proof it runs: Gemma 4 26B answering on the basement box, CPU-only.

I’m cleaning up the changes to send upstream as a pull request, so anyone else sitting on ancient enterprise iron can keep a local model around: a fallback for when the paid APIs are down, or a cheap way to grind through slow batch jobs when paying per token doesn’t make sense.

For the people who want the actual bug

Full disclosure before I go further. I’m not a C++ programmer. I can read a stack trace and I know my way around a build system, but I did not hand-write kernel fallbacks for a quantized matmul engine, and I won’t pretend I did. What I did was drive. I ran the experiments, read the output, asked the next question, and knew what “correct” had to look like. The diagnosis and the patch came from the Claude instance running on the server itself. I asked it to write up what it fixed, and the rest of this section is that summary, lightly edited. If you came here from Hacker News for the real teardown, this part’s for you.

What was actually broken

The engine we needed was ik_llama.cpp, ikawrakow’s fork of llama.cpp that adds the optimizations Gemma 4’s MoE inference depends on. It assumes AVX2 as its floor. The Xeon E5-2690 v2 in this box has AVX1 but not AVX2. Turn GGML_USE_IQK_MULMAT off at build time and most of the codebase respects it: the fast paths compile out, and the model falls back to plain scalar/SSE math. That’s fine for a normal Q8_0 matmul.

Two graph ops are the exception. The Gemma 4 MoE feed-forward network emits MOE_FUSED_UP_GATE (a per-expert gate+up matmul fused with SwiGLU) and FUSED_UP_GATE (its dense analog). Both are #if-gated on GGML_USE_IQK_MULMAT inside the compute dispatcher, but the graph builder still emits them unconditionally. On this build the dispatcher’s switch had no case for those op enums, so they fell through to the default, and the destination tensors for every expert FFN silently never got computed. Gemma 4 26B has 30 layers by 8 active experts per token, so every forward pass consumed roughly 240 tensors of whatever happened to be sitting in that memory buffer already.

The symptom was fluent-looking multilingual gibberish. Token IDs spread uniformly across the 262K vocabulary, the model equally happy to emit Thai script, Korean, <unused> sentinels, or English fragments. Deterministic at temperature 0, byte-identical between single- and multi-threaded runs, no NaNs anywhere. Just a hidden state getting shoved by a large constant every layer until the final softmax went flat.

That determinism is what cracked it. Claude instrumented the raw logits before sampling, printing the top-5 tokens plus range, mean, and NaN count. The numbers gave it away: a mean logit of +16 for the first predicted token when it should sit near zero, and about 80% of the vocabulary at positive logits. Random corruption doesn’t look like that. A bias that clean only happens when a big chunk of the hidden state is uninitialized memory that happens to hold small positive floats.

The fix

Two commits on top of the fork’s main.

1. Compile fixes. The scalar #else branches for quantize_row_q8_0_x4 and quantize_row_q8_1_x4_T in iqk_quantize.cpp weren’t actually scalar. They still referenced hsum_i32_8 and other AVX2 helpers. Those got rewritten as portable scalar loops, with #if GGML_USE_IQK_MULMAT guards added around a handful of stray IQK calls leaking through ggml.c and ggml-quants.c, plus a missing include so iqk_cpu_ops.cpp compiles standalone. Without these, the fork won’t build at all on non-AVX2 hardware.
2. The runtime bug. Rather than touch the dispatcher, the fix makes the graph builder emit ops that do have compute paths on this build. In ggml_moe_up_gate, when GGML_USE_IQK_MULMAT is off: if the weight is the combined up_gate_exps tensor (shape [n_embd, 2*n_ff, n_experts], gate in the first half, up in the second), split it into two ggml_view_3d slices, run two separate ggml_mul_mat_id calls, and combine them with ggml_fused_mul_unary(gate, up, SILU). If gate and up are already separate weights, skip the split and do the same two mul-mat-IDs plus the fused mul-unary. ggml_fused_up_gate, the dense version used in non-MoE layers, gets the same treatment. Every op involved already has a working non-IQK implementation (mul_mat_id is stock ggml, and fused_mul_unary does the SILU-and-multiply in one pass). The whole change sits behind #if !GGML_USE_IQK_MULMAT, so an AVX2 build stays bit-identical to what it was before.

The fallback costs something, two separate matmul-IDs instead of one fused kernel, but this CPU is memory-bandwidth-bound anyway, and the fused kernel was AVX2-only, so we weren’t giving anything up. End to end we get about 5 tok/s decode on a 26B-A4B MoE.

One more gotcha. --run-time-repack reorders quantized weights into an AVX2-only interleaved layout (Q8_0_R8) at startup, which garbles output on AVX1 the same way. That’s a separate bug, and the patch doesn’t try to fix it. The run script just drops the flag.

The instruction-set mismatch was easy to spot. The silent fall-through was not. Reading the code kept clearing the obvious suspects: the RMSNorm helpers looked correct, the AVX1 fallback in ggml_vec_dot_q8_0_q8_0 looked correct, and a bit-identical single-thread run ruled out threading. Only after instrumenting the logits, and seeing the mean pinned at +16 with every long-tail token roughly tied, did the search narrow to “a big chunk of the residual stream is uninitialized.” Grepping for #if GGML_USE_IQK_MULMAT in the dispatcher turned up the two missing cases about a minute later.

Why this is on the company blog

“Good with AI” has started to mean “pays for a subscription.” What we actually sell is narrower and more useful: knowing these systems well enough to aim them at a hard problem and recognize when they’ve solved it. It’s the same judgment we bring to a fifteen-year-old Rails app, or a database nobody left on the team understands. The value is knowing where the leverage is, and what the tool won’t tell you on its own.

If you want people like that looking after your software, we’d love to talk.

---

A couple of pointers for the curious. The server itself cost under $300; here’s the math on why a basement box beats $1,500 a month of cloud. And getting a screaming enterprise appliance quiet and bootable in the first place was its own project, written up here for people who enjoy that sort of thing.