Run AI Locally: Ollama, LM Studio, Best Models

A local LLM is a large language model whose weights you download and run on your own computer, instead of sending prompts to a cloud API. After the initial download, no internet connection is required, no API key is needed, and no per-token fee is charged. Your prompts and responses never leave your machine, which is a decisive advantage for healthcare, legal, and proprietary code work, where data cannot leave the building — and a practical advantage for anyone who wants AI that works on a plane, in a remote office, or without a subscription.

Local models have closed much of the quality gap for everyday tasks. A well-chosen 7B to 14B model is genuinely useful for chat, coding help, summarisation, and document questions. It won't match a frontier cloud model on the hardest reasoning tasks, but for drafting and coding assistance on your own data, it's often sufficient.

This guide explains what hardware you need, which tool to pick — Ollama, LM Studio, or llama.cpp — which models to try first, and how to set it all up without downloading models that don't fit your machine.

First, the hardware rule: work backwards from memory

The most important rule for local AI: work backwards from VRAM and RAM, not from leaderboards. Your card and your memory decide which models you can run, and how fast they'll feel. A model that doesn't fit will either refuse to load, or run so slowly it's unusable.

A rough rule, from multiple 2026 guides, is that a 4-bit quantised model — a model compressed to about 4 bits per parameter, to shrink its size — needs about 0.6 to 0.7GB of memory per billion parameters. So:

• A 7B model needs roughly 5 to 6GB.
• A 13B model needs roughly 9 to 10GB.
• A 70B model needs roughly 40GB, in quantised form, and about 140GB at full precision — which no consumer machine has.

Add a few gigabytes of headroom for your operating system and for the context window — the amount of conversation and document text the model can hold at once. Longer context needs more memory, which is why a model that fits for a short chat may struggle with a long document.

Here's a practical table, synthesised from October 2026 guides:

Apple Silicon Macs are especially strong for local AI, because the GPU shares unified memory with the CPU. A 16GB M-series Mac handles mid-size models well, and a Mac with 64GB can run some quantised 70B models at 5 to 15 tokens per second, according to guides. On Windows and Linux, an NVIDIA GPU with 6GB or more VRAM gives a much smoother experience than CPU alone, and an RTX 3060 with 12GB or an RTX 4060 Ti with 16GB handles 7B to 13B models comfortably at 20 or more tokens per second.

Ollama, LM Studio, or llama.cpp: which tool should you use

Three tools cover almost all local AI use cases, and they all run the same GGUF model files, built on llama.cpp underneath. The choice is about interface, not capability.

Ollama: best for developers

Ollama is a lightweight command-line tool that makes managing local models feel as simple as using Docker. It's MIT-licensed, with roughly 179,000 GitHub stars in October 2026 coverage, and it now ships a desktop GUI alongside the CLI.

Why developers like it:

• One command to pull and run a model: ollama run qwen3.5:9b (model names vary by library listing — check the Ollama library for the exact tag).
• An OpenAI-compatible endpoint on localhost, so existing tools and scripts can point at your local model with minimal changes.
• Anthropic Messages API compatibility added in January 2026, which is how people point Claude Code at a local model, according to StorageReview's guide.
• Automatic VRAM detection, with a warning if memory is short.

Minimum hardware, per the same guide: 8GB system RAM for a 3B model, 16GB for 7B to 8B, 24GB VRAM for the 30B class. Ollama supports Windows 10 22H2 and later, and macOS 14 and later, including Intel Macs on CPU only.

LM Studio: best for exploring and chatting

LM Studio is a full desktop application with a graphical interface. No command line required. You browse models in a visual catalogue, download directly from Hugging Face, chat through a clean UI, and control inference parameters like temperature, top-p, and context length with sliders.

Why non-developers like it:

• No terminal, no commands, no configuration files to edit by hand.
• A built-in chat interface, plus a local server on port 1234 with an OpenAI-compatible API, so you can still connect other apps to it.
• Fine-grained control over quantisation — choose Q4, Q5, or Q8 formats depending on how much VRAM you have to spare.

LM Studio is closed source, but free for commercial use, and it bundles both llama.cpp and MLX (Apple's machine learning framework) under the hood. On Mac, it requires Apple Silicon. Minimum hardware is 16GB RAM, or 24GB VRAM, or 32GB unified memory for larger models, per StorageReview.

Many developers use both: LM Studio for exploring models and testing prompts visually, Ollama for production workflows, scripts, and coding tools. There's no penalty for installing both — they share the same model formats, and you can switch between them as your task changes.

Which models to try first, by task

Model choice matters more than tool choice, and changes as new models are released. Here are solid starting points from October 2026 guides, organised by what you actually want to do. Always check the model card for the exact tag, size, and licence before you download — licences vary, and they matter for commercial use.

General chat and writing

• Llama 3.3 8B: the safe general default. Fast, good across conversation, writing, and code, and runs on any setup with 8GB or more RAM. Meta's Llama family is the most widely deployed open-weight family, with a huge ecosystem of fine-tunes.
• Gemma 4 12B: Google's efficient, lightweight family, designed to run well on laptops. Good general fallback, with multimodal variants in some sizes. Runs on 12 to 16GB GPU.
• Qwen 3.5 9B: Alibaba's Qwen family consistently tops open-model leaderboards for reasoning, multilingual work, and coding. The 9B size is a sweet spot for 16GB machines.

Coding

Coding is the most common reason people run a model locally, and it's where model choice matters most.

• Qwen3-Coder 30B: a 30B mixture-of-experts coding model, in a 17 to 20GB package depending on quantisation, with a large context window. Fits on a 24GB GPU or 32GB unified Mac. Strong for repository-scale work.
• Qwen3.6-27B: cited in October 2026 coding comparisons as the best local coding model for a 24GB-class machine, with a 256K token context window, in a 17GB Ollama package, standing out for repository-scale reasoning and agentic tool use.
• Devstral Small 2: Mistral's 24B model, built for multi-file agentic engineering tasks, fitting on a 32GB Mac or RTX 4090, with about 128K context.
• gpt-oss 20B: OpenAI's open-weight mixture-of-experts model, a 14GB download that needs about 16GB of memory, with a ChatGPT-style feel fully offline. Good for developers who want an OpenAI-style model without an API.
• Gemma 4 12B and Qwen3.5-9B: good coding options on smaller machines, for edits and smaller files, where a 30B model won't fit.

A few habits get the most from any coding model: default to Q4_K_M quantisation, and step up to Q5 or Q6 only with spare VRAM; verify the licence before commercial use — Qwen and gpt-oss use permissive Apache 2.0, while Llama and Gemma carry their own community terms; and pair a local model with a human review step for anything that ships to production. Local models are excellent for drafting and for first-pass review, but they still hallucinate, and they still need checking.

Setup in five steps

Here's a minimal, practical setup using Ollama. LM Studio follows the same logic, with clicks instead of commands.

1. Check your hardware. Note your RAM and VRAM, and pick a model size from the table above that leaves 2 to 4GB of system memory overhead. Don't download a 30B model on an 8GB laptop and wonder why it crawls.
2. Install Ollama from ollama.com, or LM Studio from lmstudio.ai. Ollama needs about 350MB for the base install, plus model files. LM Studio needs a larger download, but gives you a GUI.
3. Pull a model. In Ollama, that's a single command, for example ollama run qwen3.5:9b — check the library for the exact tag for the size you want. In LM Studio, search the catalogue, download, and click Load.
4. Chat and test. Ask the model to summarise a document you already know well, or to explain a piece of code you understand. That gives you a baseline for quality, speed, and hallucination rate, before you trust it with real work.
5. Connect a tool, if you want to. Ollama's localhost endpoint works with VS Code plugins like Continue, Cline, Tabby, and Aider, and with Open WebUI for a ChatGPT-style interface in your browser, all running locally. LM Studio's local server on port 1234 does the same.

Privacy: what local really means, and what it doesn't

Local AI is private in a specific, important sense: your prompts are not sent to a third-party API, there are no server logs of your conversations, and no third-party data processing agreement is needed. For regulated work, that's a game-changer.

The honest limits

Local AI in 2026 is genuinely useful, but it's not a replacement for frontier cloud models for everyone, and it's worth stating the limits plainly.

First, capability. A 7B to 14B local model is good, but it's not GPT-6, Claude Opus, or Gemini at their best, especially for complex reasoning, long-horizon agent tasks, and cutting-edge coding. If you need the absolute best model for a critical task, use the cloud, and use local for everything else.

The verdict

Run AI locally if you value privacy, offline access, predictable cost, and control over your data. Start with Ollama or LM Studio, pick a model that fits your RAM with room to spare, and test it on work you already understand. For most people on a 16GB laptop, a 7B to 12B model — Qwen, Gemma, or Llama — is the sweet spot: fast enough to be useful, small enough to run well, and free after the download.

The best setup in 2026, for many developers, is both: a local model for everyday work on your own data, and a cloud model for the tasks that genuinely need it. That gives you privacy where it matters, capability where it counts, and no single point of failure when the internet goes down or an API changes its pricing. And it starts, as all good local setups do, with one download, one model, and a machine you already own.