How much VRAM do I need to run a 70B model locally?

For a Q4_K_M quantized 70B model, you need roughly 40GB of VRAM to comfortably load the weights and maintain an 8k context window. This typically requires dual 24GB GPUs like RTX 3090s or 4090s.

Why compile llama.cpp from source instead of using Ollama?

Compiling llama.cpp from source allows you to enable native CPU instructions like AVX2 or AVX-512, and optimal CUDA support for your specific hardware. Pre-compiled binaries often leave significant token generation performance on the table.

What is the best cloud provider for testing large LLMs?

For dedicated hardware, Hetzner offers excellent GPU pricing well below AWS or DigitalOcean. For temporary, hourly inference testing, RunPod is highly recommended due to low per-hour costs for A100 or RTX 4000 instances.

Surviving the LLM Drop Season: A Practical Guide to Running the New Beasts

Table of Contents

I’m exhausted. Between Qwen 2.5, Llama 3.2, and whatever Mistral just dropped, keeping up with r/LocalLLaMA this week feels like drinking from a firehose. Someone in the daily thread literally posted: “Me: Worn out from all the new model drops this week, but still hyped for all the great new releases.”

Relatable. But being hyped doesn’t mean you need to fry your hardware trying to run a 70B model on a laptop. Here is how to actually deploy these new releases without losing your mind.

Step 1: Ditch the upstream Python packages #

I love Ollama, but downloading the standard installer is a trap. You are leaving performance on the table.

If you want the actual speed the community is hyping about, you need llama.cpp compiled locally for your exact CPU. AVX2 or AVX-512 matters.

Grab the latest source and build it. It takes two minutes.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make GGML_NATIVE=1

If you have an NVIDIA card, swap that make command for make GGML_CUDA=1 and thank me later. Running pre-compiled binaries is why your tokens per second look like a slideshow.

Step 2: Provision a real GPU box (without going broke) #

Running a 70B model locally requires roughly 40GB of VRAM in standard Q4_K_M quantization. That means dual 3090s. I haven’t tested this on ARM yet, and quite frankly, if you are trying to run dense 70Bs on a Mac Studio, the unified memory bandwidth is going to bottleneck you anyway.

Your best bet for testing these massive drops on a budget is cloud hosting.

Skip AWS. Skip DigitalOcean. Go to Hetzner. You can grab a dedicated GPU box (like a barebones rig with an RTX 4000 or 3090) for under a hundred bucks a month. You can even share an A100 node via RunPod for about a dollar an hour.

Step 3: Spin up the server #

Forget Docker. For inference overhead, Podman is the move here. Rootless containers mean you don’t accidentally expose your model weights to the web.

Fire up an Ubuntu 22.04 box, install the NVIDIA container toolkit, and pull the image.

sudo apt-get install -y podman
podman run --device nvidia.com/gpu=all \
  -p 8080:8080 \
  -v /path/to/models:/models \
  ghcr.io/ggerganov/llama.cpp:server \
  -m models/Qwen2.5-72B-Instruct-Q4_K_M.gguf \
  -c 8192 --host 0.0.0.0 --port 8080

That -c 8192 sets the context window. Closer to 32k will eat 6GB of VRAM just for KV cache. Keep it reasonable unless you are summarizing entire codebases.

Step 4: Pick the right backend (and the right quant) #

The llama.cpp repo changes daily.

One fatal flaw with the bleeding edge releases: GGUF quants sometimes break. A Q6_K might throw a tensor error on Tuesday and get fixed by Thursday. If it crashes, drop to Q4_K_M. It is the community gold standard for a reason.

I saw a post in the daily thread complaining about Llama 3.2 running hot and spitting garbage. Don’t use the default Ollama pull. The default quant files often strip essential high-precision tensors. Grab your GGUFs directly from Bartowski on Hugging Face. Trust the community quants, not the automated pipeline.

Ground rules for sanity #

Do not run everything.

This is overkill for most people, but I have a local scrap server running a 7B model via vLLM just for writing bash scripts. If I need to review a giant PR or parse a 50-page PDF, I spin up the Podman container, run the 70B, and kill it when I’m done.

Your mileage may vary depending on your setup, but keeping the 70B heavy hitters ephemeral saves your SSD and your power bill. Let the models drop, wait a day for the quants to catch up, and then actually build something.