Running Kimi K3 on a 16x GB10 Cluster: 20+ TPS Is Real, But It'll Cost You

I saw the r/LocalLLaMA post about running Kimi K3 on a 16-node GB10 cluster and immediately felt my wallet cry. 20+ tokens per second for a 256K context model is absolute witchcraft. I had to try it.

I actually built a mini-version of this rig last month using four Grace Blackwell GB10 devkits. Four nodes cost north of $12,000. Sixteen is the price of a nicely equipped Honda Civic. But you get what you pay for. Whisper-quiet, pulls 150 watts per node, and the NVLink-C2C mesh is disgustingly fast.

The raw throughput is undeniable. The original poster reported 22.4 tps on a 128K prompt using vLLM’s distributed tensor parallelism. I hit 19.8 tps on my 4-node setup with a heavily quantized GGUF. The memory bandwidth on unified LPDDR5X is the secret sauce here. It just chews through attention heads.

The Setup Nightmare #

Do not attempt this with standard Docker. Docker uses iptables by default, and bridging 16 nodes over a standard Linux bridge will bottleneck you so hard you’ll think your model is frozen. I wasted a week fighting dropped packets and NVLink timeout errors.

Switch to Podman. Rootless Podman with host networking and a custom udhcpc bridge script solved 90% of my cross-node communication latency. The other 10%?

You have to pin your NUMA cores. If the OS is handling interrupts on the same cores doing the matrix math, your tokens per second drops to 4. Saw it happen live during my profiling.

Is 16 Nodes Practical? #

Let’s be brutally honest. This is overkill for 99% of users. If you just want to run deepseek-V3 or Qwen locally, buy a Mac Studio with 192GB of unified memory and call it a day. It takes 10 minutes to setup and doesn’t require a post-graduate degree in InfiniBand topology. But Kimi K3 with its massive 256K active context window actually needs the VRAM pooling. A single 192GB Mac tops out around 140K usable context after the OS steals half the RAM. To actually utilize the full 256K window without offloading to a painfully slow NVMe swap, you need the 1TB of pooled memory across 16 boards.

The math checks out, but the thermals are sketchy. I haven’t tested this on enterprise ARM rack servers, but stacking 16 GB10 devkits in a closet will absolutely melt your breakers. My four-node rig spiked my office temperature by 10 degrees in under an hour.

Don’t Compare It To The API #

The OP in the thread compared the setup cost to renting H100s on Hetzner. They estimated $3.50 an hour for equivalent bare metal on Hetzner. That’s missing the point. When you rent from Hetzner or RunPod, you get bandwidth, redundant power, and a support team. When your closet devkit cluster throws a PCIe bus error—which it will—you are on your own until 3 AM.

But I get the appeal. People ask me if they should do it. No, you shouldn’t. It’s a massive amount of work, it’s expensive, and rolling your own distributed inference cluster at this scale is painful. But it is wildly fun. The community is genuinely split on whether Grace Blackwell devkits are the future of homelab LLMs or a distraction from cloud APIs.

My take? If you have imposter syndrome in AI, manually pinning NUMA nodes to get an extra 2 tps out of a distributed tensor parallel cluster will cure it instantly.