Table of Contents

Someone on r/LocalLLaMA actually asked about pushing Kimi K3 onto a single CPU with 8 GB of RAM. I read the title, laughed, and then realized they were dead serious.

I run a lot of local models. I have a 64GB Linux tower specifically for this garbage. But I love a stupid challenge, so I kicked my M1 Macbook into sleep mode, dusted off a 4-core Intel N100 mini-PC with 8 GB of RAM, and decided to see exactly how badly I could break a modern model.

Spoiler: badly, but not in the way you think.

The 8GB Ceiling #

Let’s be brutally clear about hardware. A 4-bit quantized MoE model like Mixtral eats around 24 GB of RAM just to load into memory. Kimi K3 is a massive 256B/12B active MoE. Even heavily quantized to 3-bit, you are looking at roughly 90 GB of weights. On an 8 GB machine, the operating system is already eating 2 GB just to keep the lights on. That leaves you 6 GB.

When you try to load a 90 GB file with 6 GB of physical RAM, your OS relies on swap memory. It starts reading weights from your SSD instead of your RAM. Your CPU is now waiting on your disk bus. Throughput drops to literally 0.2 tokens per second. You can read War and Peace before the model finishes its first sentence. As user u/synth_local pointed out in the thread, “at that point you are just emulating an abacus.”

The Groq vs Hetzner Pivot #

If you are stuck on an 8 GB machine, running Kimi K3 locally is a pure masochism play. You need compute elsewhere. Let’s look at the actual pragmatic alternatives the community agrees on.

You spin up a Hetzner box. The CCX33 instance gives you 8 dedicated AMD vCPUs and 32 GB of RAM for around $0.05 an hour. You can install llama.cpp via a simple Docker container, fire up an SSH tunnel, and stop crying. Even then, 32 GB is hopelessly fatal for the full K3 model. If you absolutely must run MoE architectures locally without burning your house down to power dual 3090 GPUs, your best bet is a smaller release. Grab the Mistral 7B or Qwen 2.5 14B GGUF files from Bartowski. Those will load perfectly inside 8 GB, leave room for a decent context window, and actually output text at a readable 8 to 12 tokens per second.

Docker vs Podman on the Potato #

If you are remotely managing a local build on a system this weak, your container overhead actually matters.

Docker is the standard, and almost every LLM repo on GitHub assumes you are running it. Docker Desktop on an 8GB Windows machine is a fast track to the blue screen of death.

Here is the fix: install Podman. I haven’t tested this on ARM extensively yet, but on x86 it runs rootless and saves you about 300 MB of idle RAM overhead. 300 MB isn’t a lot for my big rig, but on an 8 GB potato, that is the difference between your OS aggressively swapping and the model actually staying pinned in active memory.

Don’t Fight the Hardware #

I love local inference, but pretending hardware limits don’t exist is just coping. The community is genuinely split on how to handle the 8GB requirement.

One camp swears by the Oracle Cloud free tier, loading a 4-bit model onto an Ampere A1 ARM instance. It is free, but you get throttled into oblivion and the compute instances are notoriously scarce. Another camp just pays for an API key from OpenRouter, DeepSeek, or Anthropic and calls it a day. For $5, you can run Kimi K3’s cloud API for a month without even thinking about thermals or swap memory. Your mileage will absolutely vary depending on your tolerance for CLI debugging. But if you only have 8 GB of RAM, respect the ceiling.

Run Qwen. Be happy.