Running LLMs Locally: A Practical Guide for Developers
A year ago, running a capable language model on your own machine was a research project. Today it's a Tuesday afternoon. Models like Llama, Mistral, Gemma, and Phi have gotten good enough that local inference is a serious option for many use cases — not just a party trick.
Here's when it makes sense and how to set it up properly.
When local beats cloud
Privacy-sensitive data. If your app processes medical records, legal documents, financial statements, or anything your users wouldn't want leaving their machine, local inference removes the API call from the threat model entirely.
High-volume, low-latency tasks. For tasks where you're making thousands of small calls — document classification, entity extraction, code linting — cloud API costs add up fast. A local model has zero marginal cost per call.
Offline or air-gapped environments. Some industries (defense, healthcare, finance) have environments that simply can't reach the open internet. Local models are the only option.
Development and experimentation. Iterating on prompts with a local model means no rate limits, no latency, no cost per call. The iteration loop is dramatically faster.
The tooling has gotten good
Ollama is the easiest way to run models locally. It handles model downloading, hardware detection, and exposes a simple REST API that's compatible with the OpenAI client:
# Install and pull a model
ollama pull llama3.2
ollama pull qwen2.5-coder # great for code tasks
# It's running on localhost:11434
from openai import OpenAI
# Drop-in replacement — just change the base URL
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
model="llama3.2",
messages=[{"role": "user", "content": "Explain async/await in Python"}],
)
print(response.choices[0].message.content)
If you have existing code using the OpenAI client, switching to a local model is literally two lines.
Picking the right model for your hardware
Model size determines hardware requirements. A rough guide:
| Model Size | RAM Required | Good For | |-----------|-------------|----------| | 1–3B | 4–6 GB | Simple classification, short responses | | 7–8B | 8–10 GB | General tasks, most development use cases | | 13–14B | 16 GB | Better reasoning, longer context | | 32B+ | 32+ GB | Near GPT-4 quality for complex tasks |
If you have a modern Mac with unified memory, you can run surprisingly large models. An M3 Pro with 36GB handles 14B models comfortably.
For most development tasks, a 7–8B model at 4-bit quantization hits the sweet spot: fast enough to feel interactive, good enough for most tasks, runs on a decent laptop.
Quantization: quality vs. speed
Full-precision models (FP16) are the most accurate but also the largest. Quantized models trade a small accuracy drop for dramatically smaller size and faster inference.
In practice:
- Q8 — nearly identical to FP16, only 2x smaller. Prefer this if you have the RAM.
- Q4_K_M — the sweet spot. 4x smaller than FP16 with minimal quality loss for most tasks.
- Q2 — very small but noticeably worse. Avoid unless hardware is extremely limited.
Ollama handles quantization automatically when you pull a model. The model card will tell you which quantization level it uses.
When not to go local
Local inference has real limits:
- Top-tier reasoning (complex coding, multi-step analysis) still favors frontier models
- Context windows are smaller — most local models top out at 8–32K tokens
- Setup and maintenance require technical knowledge your users may not have
- Multi-user serving is complicated without proper infrastructure
For anything requiring the best possible output quality, cloud APIs remain the right call. Local inference is about the right tool for the right job — not replacing cloud AI wholesale.
The practical move: use local models for development iteration and high-volume low-stakes tasks. Use cloud APIs for anything requiring frontier-level reasoning or where output quality is critical. The gap between local and cloud is narrowing every quarter.