Local LLMs in 2026: run a capable model on a laptop

Two years ago, “local LLM” meant a slow, hallucination-prone toy on a gaming PC. In 2026 it means a 30-billion-parameter quantised model answering emails on a MacBook Pro while you are on a train with no signal. The shift is not magic — it is better architectures, aggressive quantisation and unified memory on Apple Silicon and modern NVIDIA GPUs. The question is no longer whether you can run a model offline, but which stack matches your workload and your laptop’s thermals.

I tested local setups on three machines: a 16GB M3 MacBook Air, a Ryzen laptop with an RTX 4060 (8GB VRAM), and a desktop with 64GB RAM and an RTX 4080. The goal was real work — summarising PDFs, drafting code, rewriting meeting notes — not leaderboard scores. What follows is the stack that survived that grind.

Choose your runtime: Ollama vs llama.cpp vs LM Studio

Ollama is the default recommendation for most people. Install one binary, pull a model with ollama pull, and you have an OpenAI-compatible API on localhost. It handles model downloads, GPU layer offloading and updates without drama. Developers integrating local inference into apps should start here.

llama.cpp is the engine underneath many GUIs — lower level, more knobs. Use it when you need custom quantisation, CPU-only inference on a machine with no discrete GPU, or embedding generation with tight memory budgets. LM Studio and Jan.ai wrap llama.cpp with friendly interfaces for non-terminal users. Pick Ollama for servers and scripts; pick a GUI if you live in chat windows.

Which models actually fit on a laptop

Model choice is a memory equation. A Q4_K_M quant of Llama 3.3 70B needs roughly 40GB — feasible on a maxed MacBook Pro or a desktop, not on 16GB ultraportables. For 16GB machines, look at 8B–14B class models: Mistral Nemo, Qwen2.5 14B, Phi-4, Gemma 2 9B. They handle summarisation, light coding and drafting competently at 20–40 tokens per second on Apple M3.

The surprise winner on the RTX 4060 laptop was a 32B Q4 model with partial GPU offload — slower than cloud GPT-4o, but usable for offline contract review on a client site where USB sticks are allowed and Wi-Fi is not. Always check context length: 128k context at Q4 eats RAM faster than the model weights alone. For long documents, prefer models with native 32k and chunk your inputs.

“Open-weight models closed much of the capability gap with proprietary APIs in 2025 — the remaining difference is less raw IQ and more integration, tooling and safety tuning.”

— Meta AI research blog on Llama 4, reported by BBC Technology

Hardware: what to buy and what to skip

Apple Silicon remains the efficiency king for local inference. Unified memory means a 36GB M3 Max runs models that would choke a 16GB Windows laptop with a discrete GPU. If you are buying for local AI on the road, RAM matters more than CPU cores — 24GB minimum for comfortable 14B models, 36GB+ for 32B experiments.

On Windows, NVIDIA still wins for raw throughput. AMD’s ROCm support improved but remains finicky on laptops. Intel’s Arc GPUs are improving for llama.cpp via SYCL backends — relevant if you read our AMD vs Intel piece and chose integrated Xe for a thin corporate machine. CPU-only inference is viable for 7B models on recent Ryzen chips; it is not viable for interactive 70B dreams.

Privacy, compliance and when local wins

Law firms, NHS contractors and defence-adjacent suppliers often prohibit cloud inference on client material. Local models keep data on disk you control. Air-gapped installs — download weights on a networked machine, sneakernet to the secure laptop — are clumsy but approved in environments where even private VPC endpoints fail audit.

The trade-off is responsibility: no vendor SOC 2 report, no automatic safety filters tuned for consumer harm. You must patch models, rotate API keys on localhost services exposed to LAN, and accept that open weights can be fine-tuned without alignment. For internal summarisation on trusted docs, that is fine. For customer-facing chatbots, use cloud tiers with contractual guarantees.

Integration patterns that work

Most productivity gains come from wiring localhost:11434 into tools you already use. Obsidian plugins, VS Code Continue extension, and custom Python scripts using the OpenAI SDK with base_url="http://localhost:11434/v1" are common patterns. RAG — retrieval-augmented generation — improves factual answers: embed documents with nomic-embed-text locally, store in Chroma or LanceDB, inject chunks at query time.

Avoid running a local model and a cloud model in the same thread without labelling outputs. Teams report confusion when half a document was drafted offline and half was polished online — subtle style shifts and fact drift compound.

Tuning, frustration and realistic expectations

Quantisation hurts maths and structured extraction more than prose. If you need reliable JSON from invoices, test Q8 quant or a smaller full-precision model. Temperature 0.2 for extraction; 0.7 for brainstorming. System prompts matter more on small models — be explicit about format and refusals.

Local LLMs will not replace GPT-5 for agentic coding or multimodal analysis in 2026. They will replace sending your NDA draft to a US API because you forgot to check the policy. That is a worthwhile win.