Local LLM Inference for Coding: The Complete 2025/2026 Guide

Running AI coding assistants locally has shifted from a hobbyist experiment to a genuinely competitive alternative to cloud-based tools. In 2025, models like Qwen2.5-Coder-32B match GPT-4o on HumanEval benchmarks, and the tooling — Ollama, LM Studio, llama.cpp — has matured to the point where setup takes minutes rather than days. This guide covers everything from model selection and hardware requirements to full editor integration with Continue.dev and Aider.

Part 1: The Best Local Models for Code Generation

The Benchmark Landscape

Three benchmarks matter most for evaluating coding models:

• HumanEval: 164 Python function-completion problems. A widely-used but increasingly criticized benchmark because models can be trained to game it.
• MBPP (Mostly Basic Python Problems): 500 crowd-sourced Python tasks. Broader coverage than HumanEval.
• SWE-bench: Real GitHub issues that must be resolved by modifying actual codebases. Far harder and more representative of real engineering work. Even frontier models struggle here.
• LiveCodeBench: Ongoing competitive programming problems with a time-cutoff to prevent data contamination.

Model-by-Model Breakdown

Qwen2.5-Coder (Alibaba) — The Current Leader

Alibaba’s Qwen2.5-Coder family dominates every size tier in 2025/2026. Trained on over 5.5 trillion tokens of code-heavy data, it supports 40+ programming languages and ships in six sizes (0.5B to 32B). Critically, all sizes support Fill-in-the-Middle (FIM), which is essential for IDE autocomplete.

The 32B model at Q4_K_M quantization fits in approximately 20GB of VRAM, making it viable on a single RTX 3090 or 4090. For autocomplete, the 1.5B model runs on almost any hardware and has low enough latency to feel instant.

DeepSeek-Coder-V2 — MoE Architecture and 128K Context

DeepSeek-Coder-V2 uses a Mixture-of-Experts architecture and was the first open-source model to break 10% on SWE-bench, which represents a genuine milestone for real-world coding tasks.

The Lite variant is the practical local choice: because only 2.4B parameters activate per inference, it is fast and memory-efficient while delivering strong results. The full 236B model is impractical for single-GPU consumer hardware but can be used via API.

Training data was 10.2 trillion tokens: roughly 60% source code across 338 programming languages, 10% mathematics, and 30% natural language.

CodeLlama (Meta) — Aging But Accessible

CodeLlama is based on Llama 2 and continues pre-training on 500–1,000 billion code tokens. It comes in four sizes: 7B, 13B, 34B, and 70B. While it was a dominant model in 2023, it has been significantly outpaced by Qwen2.5-Coder and DeepSeek.

CodeLlama is still useful when VRAM is severely constrained, since it is widely available in GGUF format at many quantization levels. Its instruction-following variants (CodeLlama-Instruct) are better for chat-style use.

StarCoder2 (BigCode / Hugging Face) — Open Training Data

StarCoder2 was trained on The Stack v2 dataset covering 619 programming languages. Its key advantage is a fully transparent, open training data provenance — important for organizations with compliance requirements.

StarCoder2-15B at Q4_K_M is roughly 10GB and is a strong autocomplete model. It was not trained to game HumanEval specifically, meaning its CRUXEval (code execution reasoning) scores are more correlated with its HumanEval scores — a sign of genuine capability rather than benchmark overfitting.

WizardCoder — Instruction Tuning via Evol-Instruct

WizardCoder applies the Evol-Instruct method to StarCoder and CodeLlama base models, iteratively transforming training instructions to be more complex and diverse.

WizardCoder-34B surpassed GPT-3.5 on HumanEval+ (73.2% vs 63.4%). However, like many instruction-tuned models, it may not generalize as broadly as its benchmark scores suggest, and it has largely been superseded by Qwen2.5-Coder.

Mistral and Mixtral for Coding

Mistral’s Codestral-22B is purpose-built for code, scoring 81.1% on HumanEval and including strong FIM support. The Mixtral MoE variants (8x7B, 8x22B) are capable general-purpose models that handle coding reasonably well, though they are outperformed by Qwen2.5-Coder at equivalent compute budgets. Mixtral 8x7B activates about 12B parameters per inference, fitting in roughly 24GB of VRAM for the Q4 variant.

Comprehensive Benchmark Comparison Table

Part 2: Ollama

Ollama is the simplest way to run LLMs locally. It handles model downloads, quantization selection, server lifecycle, and exposes both a native REST API and an OpenAI-compatible API at http://localhost:11434.

Installation

1
2
3
4
5
6
7
8

# macOS (Homebrew)
brew install ollama

# Linux (one-liner installer)
curl -fsSL https://ollama.ai/install.sh | sh

# Windows
# Download the installer from https://ollama.ai

Verify the installation and start the service:

1
2
3

ollama --version
ollama serve  # starts in background on macOS/Windows automatically after install
curl http://localhost:11434  # should return: Ollama is running

Pulling Coding Models

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Best all-round coding model if you have 24GB VRAM
ollama pull qwen2.5-coder:32b

# Strong 7B model — fits in 8GB VRAM
ollama pull qwen2.5-coder:7b

# Fast autocomplete model — runs anywhere
ollama pull qwen2.5-coder:1.5b

# DeepSeek Coder V2 Lite — strong reasoning, efficient MoE
ollama pull deepseek-coder-v2:16b

# StarCoder2 — best for FIM/autocomplete at mid-range
ollama pull starcoder2:15b
ollama pull starcoder2:3b

# Embeddings model (needed for Continue.dev @codebase)
ollama pull nomic-embed-text

Ollama REST API

Ollama exposes two API flavors. Use the native API for full control, or the OpenAI-compatible API to drop into any existing OpenAI client.

Native Chat API (POST /api/chat):

1
2
3
4
5
6
7

curl http://localhost:11434/api/chat -d '{
  "model": "qwen2.5-coder:7b",
  "messages": [
    {"role": "user", "content": "Write a Python function to parse JWT tokens without a library."}
  ],
  "stream": false
}'

Native Generate API (POST /api/generate):

1
2
3
4
5

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:7b",
  "prompt": "def fibonacci(n):",
  "stream": false
}'

List local models (GET /api/tags):

1

curl http://localhost:11434/api/tags

OpenAI-Compatible Chat Completions (POST /v1/chat/completions):

1
2
3
4
5
6

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "messages": [{"role": "user", "content": "Explain async/await in Python"}]
  }'

This OpenAI-compatible endpoint lets you point any OpenAI SDK client at Ollama by changing base_url:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by the client, value ignored
)

response = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role": "user", "content": "Write a Rust function to reverse a string"}]
)
print(response.choices[0].message.content)

Model Library

Ollama’s model library at ollama.com/library lists all available models. Key coding models available as of early 2026:

• qwen2.5-coder — 0.5b, 1.5b, 3b, 7b, 14b, 32b
• deepseek-coder-v2 — 16b, 236b
• starcoder2 — 3b, 7b, 15b
• codellama — 7b, 13b, 34b, 70b
• deepseek-r1 — 1.5b, 7b, 8b, 14b, 32b, 70b, 671b
• llama3.1, llama3.2, llama3.3 — general-purpose with strong coding
• mistral, mixtral — capable general-purpose models

VS Code Integration

Ollama has a first-class VS Code integration page at docs.ollama.com/integrations/vscode, and is the recommended backend for the Continue.dev extension (see Part 6).

For a remote Ollama server, set the host environment variable before starting the service:

1

OLLAMA_HOST=0.0.0.0 ollama serve

Then connect clients to http://your-server-ip:11434.

Part 3: LM Studio

LM Studio is a polished desktop GUI application that handles local model management, inference, and serves an OpenAI-compatible API. It targets both developers and non-technical users, and is available for macOS, Windows, and Linux.

Key Features

• Model Browser: Search and download models directly from Hugging Face within the app. Filter by size, architecture, and quantization level.
• OpenAI-Compatible Server: Exposes http://localhost:1234/v1 with chat completions, embeddings, and — as of v0.3.29 — the OpenAI Responses API including stateful interactions (previous_response_id), function tool calling, remote MCP server connections, reasoning effort controls, and streaming.
• Multiple Inference Engines: Ships with both llama.cpp (for GGUF models) and Apple MLX (for Apple Silicon Macs). The engine is selected automatically based on hardware.
• Hardware Detection: Automatically detects GPU capabilities and configures layer offloading. Supports Vulkan for integrated Intel/AMD GPUs.
• Intelligent Memory Management: Can split models across GPU VRAM and system RAM. Note: overflowing to RAM reduces speed by approximately 30x.
• Headless Daemon (llmster): LM Studio’s core packaged as a standalone daemon for server/CI deployment without the GUI.
• MCP Client: Connect external tool servers (Model Context Protocol) for agent workflows.

OpenAI-Compatible API Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="lm-studio",  # required, value ignored
)

response = client.chat.completions.create(
    model="qwen2.5-coder-32b-instruct",  # the model name shown in LM Studio
    messages=[{"role": "user", "content": "Review this Python code for bugs"}]
)

Best Models to Use in LM Studio

LM Studio’s model browser makes it easy to search Hugging Face. For coding, prioritize:

• Qwen2.5-Coder-32B-Instruct-GGUF (Q4_K_M ~20GB): Best overall local coding model
• Qwen2.5-Coder-7B-Instruct-GGUF (Q4_K_M ~5GB): Fast and capable for 8GB VRAM
• DeepSeek-Coder-V2-Lite-Instruct-GGUF (Q4_K_M ~10GB): Strong MoE reasoning
• StarCoder2-15B-GGUF (Q4_K_M ~10GB): Excellent for FIM/autocomplete

Search for models with “GGUF” in the name and filter by “instruct” variants for chat-style use.

Part 4: llama.cpp

llama.cpp is the foundational C/C++ inference library that powers Ollama, LM Studio, and many other tools. It requires no dependencies, runs on virtually any hardware, and is the reference implementation for GGUF format inference.

Compilation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# Standard build (CPU only)
cmake -B build
cmake --build build --config Release

# CUDA build (NVIDIA GPU)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Metal build (Apple Silicon — usually auto-detected)
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release

The GGUF Format

GGUF (GGML Unified Format) is llama.cpp’s native model format. It stores all model metadata — tokenizer, architecture config, special tokens, RoPE scaling parameters — in a single self-contained file. Models from Hugging Face in safetensors or PyTorch format can be converted:

1
2
3
4

pip install transformers torch

# Convert from HuggingFace format to GGUF F16
python convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf

Quantization Levels

Quantization reduces model size and speeds up inference at the cost of some quality. The K_M suffix indicates “medium quality K-quant” — these use higher precision for the most important layers.

Q4_K_M is the community standard default. Q5_K_M is recommended when you have the VRAM headroom and care about code correctness. Below Q3, quality degrades noticeably on complex coding tasks.

Quantizing a Model

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Step 1: Convert to GGUF F16
python convert_hf_to_gguf.py Qwen/Qwen2.5-Coder-7B-Instruct \
    --outfile Qwen2.5-Coder-7B-F16.gguf

# Step 2: Quantize to Q4_K_M
./build/bin/llama-quantize \
    Qwen2.5-Coder-7B-F16.gguf \
    Qwen2.5-Coder-7B-Q4_K_M.gguf \
    Q4_K_M

# Optional: Use an importance matrix for better quantization quality
# First compute the imatrix
./build/bin/llama-imatrix \
    -m Qwen2.5-Coder-7B-F16.gguf \
    -f calibration-text.txt \
    --chunk 512 \
    -o Qwen2.5-Coder-7B-imatrix.dat \
    -ngl 80

# Then quantize using the imatrix
./build/bin/llama-quantize \
    --imatrix Qwen2.5-Coder-7B-imatrix.dat \
    Qwen2.5-Coder-7B-F16.gguf \
    Qwen2.5-Coder-7B-IQ4_XS.gguf \
    IQ4_XS

GPU Offloading

Use the -ngl flag (number of GPU layers) to offload model layers to VRAM. More layers = faster inference, up to the model’s total layer count.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# CPU only (no GPU offloading)
./build/bin/llama-cli \
    -m Qwen2.5-Coder-7B-Q4_K_M.gguf \
    -p "Write a Python function to parse JSON:" \
    -n 256

# Offload all layers to GPU (CUDA or Metal)
./build/bin/llama-cli \
    -m Qwen2.5-Coder-7B-Q4_K_M.gguf \
    -p "Write a Python function to parse JSON:" \
    -n 256 \
    -ngl 99  # -ngl 99 means "as many layers as the model has"

# Partial offload — fit what you can in VRAM, rest on CPU
./build/bin/llama-cli \
    -m Qwen2.5-Coder-32B-Q4_K_M.gguf \
    -p "Write a Python function to parse JSON:" \
    -n 256 \
    -ngl 20  # offload 20 layers to GPU, rest on CPU

Partial offloading is useful when the model doesn’t fully fit in VRAM. Offloading even 50% of layers provides significant speed gains over pure CPU inference.

Server Mode

llama.cpp ships with llama-server, an OpenAI-compatible HTTP server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Basic server — binds to localhost:8080
./build/bin/llama-server \
    -m Qwen2.5-Coder-7B-Q4_K_M.gguf \
    -ngl 99 \
    --ctx-size 8192 \
    --port 8080

# Production-style server with more options
./build/bin/llama-server \
    -m Qwen2.5-Coder-7B-Q4_K_M.gguf \
    -ngl 99 \
    --ctx-size 32768 \
    --n-predict 2048 \
    --parallel 4 \        # number of concurrent request slots
    --host 0.0.0.0 \      # listen on all interfaces
    --port 8080

Connect any OpenAI client to http://localhost:8080/v1.

1
2
3
4
5
6
7

# Test the server
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Benchmarking

1
2
3
4
5
6

# Run the built-in benchmark tool
./build/bin/llama-bench -m Qwen2.5-Coder-7B-Q4_K_M.gguf

# Output includes:
# pp512  — prompt processing speed (tokens/sec)
# tg128  — token generation speed (tokens/sec)

Part 5: Hardware Requirements

VRAM Requirements by Model Size

The formula for estimating VRAM requirements is approximately:

VRAM (GB) = (Parameters in billions) × (bits per weight / 8) × 1.1

The 1.1 factor accounts for KV cache and other overhead.

Note: Larger context windows increase KV cache usage. A 32K context window on a 7B model adds roughly 1-2GB extra VRAM compared to 4K context.

Consumer GPU Recommendations

For coding specifically: an RTX 4060 Ti (16GB) at roughly $400-500 USD is the best value entry point for running Qwen2.5-Coder-7B or 14B. An RTX 3090 (24GB, used at $600-900 USD) lets you run Qwen2.5-Coder-32B, which is where local coding becomes genuinely competitive with Copilot.

CPU-Only Inference

CPU inference via llama.cpp is viable for occasional use but significantly slower than GPU:

Key insight: CPU inference is bottlenecked by memory bandwidth, not core count. Running in dual-channel RAM mode and disabling hyperthreading can improve speeds by 50-100%. High-core-count server CPUs (EPYC, Threadripper) with many memory channels dramatically outperform consumer CPUs.

For CPU-only setups, stick to 7B models at Q4_K_M (fits in ~5GB of RAM beyond OS usage). 13B models at Q4 (~8GB) are usable but slow.

Apple Silicon and the MLX Framework

Apple Silicon’s unified memory architecture gives Macs a unique advantage: the GPU and CPU share the same memory pool, so a Mac with 64GB of unified RAM can hold a 70B Q4 model entirely in “VRAM” — something impossible on a single consumer NVIDIA GPU.

MLX is Apple’s open-source ML framework optimized for Apple Silicon. Both Ollama and LM Studio ship MLX backends.

Performance benchmarks:

MLX advantages over llama.cpp on Apple Silicon:

• Models load 3x faster
• 26-30% higher token generation throughput
• Lower memory usage due to unified memory optimization
• Near-silent operation at ~50W vs ~300W+ for an RTX 4090

The M4 Pro with 24GB is the recommended minimum for serious local coding work on Apple Silicon. The M4 Max at 64GB is the sweet spot for running Qwen2.5-Coder-32B at full quality.

To use MLX with Ollama on Apple Silicon, Ollama auto-detects and uses Metal/MLX. No configuration needed.

1
2

# On Apple Silicon, -ngl 99 offloads everything to Metal/MLX
./llama-server -m Qwen2.5-Coder-7B-Q4_K_M.gguf -ngl 99

Part 6: Continue.dev with Local Models

Continue is a VS Code and JetBrains extension that provides chat, inline editing, and tab autocomplete using any LLM backend. It’s the most mature open-source Copilot alternative.

Install from the VS Code marketplace (search “Continue”) or continue.dev.

Configuration File

Continue is configured via ~/.continue/config.yaml (or the hub-based config in newer versions). The roles array controls what each model is used for.

Full Configuration Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

name: Local Coding Assistant
version: 1.0.0
schema: v1

models:
  # Primary chat model — handles complex questions, debugging, architecture
  - name: Qwen2.5-Coder 32B
    provider: ollama
    model: qwen2.5-coder:32b
    apiBase: http://localhost:11434
    roles:
      - chat
      - edit
      - apply
    defaultCompletionOptions:
      temperature: 0.1          # Low temperature for code — more deterministic
      contextLength: 32768
      top_p: 0.95

  # Fast autocomplete model — latency matters here, smaller is better
  - name: Qwen2.5-Coder 1.5B (autocomplete)
    provider: ollama
    model: qwen2.5-coder:1.5b
    roles:
      - autocomplete
    autocompleteOptions:
      debounceDelay: 200        # milliseconds to wait after keystroke
      maxPromptTokens: 1024
      multilineCompletions: auto

  # Embeddings for @codebase context provider
  - name: Nomic Embed Text
    provider: ollama
    model: nomic-embed-text
    roles:
      - embed

# Context providers give the model access to your codebase
context:
  - provider: code         # @code — search and include code snippets
  - provider: docs         # @docs — index and search documentation sites
  - provider: diff         # @diff — current git diff
  - provider: terminal     # @terminal — recent terminal output
  - provider: folder       # @folder — entire folder contents
  - provider: codebase     # @codebase — semantic search via embeddings

Alternative: Autodetect All Installed Ollama Models

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

name: Local Assistant (Autodetect)
version: 1.0.0
schema: v1

models:
  - name: Autodetect
    provider: ollama
    model: AUTODETECT
    roles:
      - chat
      - edit
      - apply
      - autocomplete

This scans your local Ollama installation and populates the model list dynamically from ollama list.

Separate Chat vs Autocomplete Models

The important design decision in Continue is to use different models for chat and autocomplete:

• Chat models (7B–32B): Handle questions, debugging sessions, and code edits. Larger is better here.
• Autocomplete models (1.5B–3B): Must respond within 300-500ms to feel instant. Qwen2.5-Coder-1.5B and StarCoder2-3B are ideal.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

models:
  # Large model for chat
  - name: DeepSeek Coder V2 Lite
    provider: ollama
    model: deepseek-coder-v2:16b
    roles:
      - chat
      - edit
      - apply
    defaultCompletionOptions:
      temperature: 0.05
      contextLength: 16384

  # Small fast model for autocomplete
  - name: StarCoder2 3B
    provider: ollama
    model: starcoder2:3b
    roles:
      - autocomplete
    autocompleteOptions:
      debounceDelay: 150
      maxPromptTokens: 2048
      multilineCompletions: auto

Using a Remote Ollama Server

If you run Ollama on a more powerful server and connect from a laptop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

models:
  - name: Remote Qwen2.5-Coder 32B
    provider: ollama
    model: qwen2.5-coder:32b
    apiBase: http://192.168.1.50:11434   # Replace with your server IP
    roles:
      - chat
      - edit
      - apply

  - name: Remote Autocomplete
    provider: ollama
    model: starcoder2:3b
    apiBase: http://192.168.1.50:11434
    roles:
      - autocomplete

Disabling Thinking Mode for Autocomplete (Qwen3)

If using a reasoning model like Qwen3 for autocomplete, disable thinking mode to keep latency low:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

  - name: Qwen3 4B (autocomplete, no thinking)
    provider: ollama
    model: qwen3:4b
    roles:
      - autocomplete
    requestOptions:
      extraBodyProperties:
        think: false
    autocompleteOptions:
      debounceDelay: 200
      maxPromptTokens: 1024

Part 7: Aider with Local Models

Aider is an AI pair programming tool that works directly in your terminal. It reads your codebase, understands context, and makes targeted edits across multiple files. It integrates with git and creates commits automatically.

Installation

1
2
3
4

pip install aider-chat

# Or with pipx (recommended)
pipx install aider-chat

Using Aider with Ollama

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Start Ollama with an extended context window
# (Ollama's default 2K context is too small for Aider)
OLLAMA_CONTEXT_LENGTH=8192 ollama serve

# In your project directory
cd /path/to/your/project

# Run Aider with a specific Ollama model
# Use ollama_chat/ prefix (recommended over ollama/)
aider --model ollama_chat/qwen2.5-coder:32b

# Or set the API base via environment variable
export OLLAMA_API_BASE=http://127.0.0.1:11434
aider --model ollama_chat/qwen2.5-coder:7b

Important: Ollama silently truncates context that exceeds its window. Aider automatically sets the context window large enough for each request plus 8K tokens for the reply, but starting Ollama with OLLAMA_CONTEXT_LENGTH=8192 (or higher) ensures this works correctly.

Custom Context Window Configuration

For models that need specific context settings, create an Aider model metadata file (.aider.model.metadata.json):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "ollama/qwen2.5-coder:32b": {
    "max_tokens": 8192,
    "max_input_tokens": 65536,
    "max_output_tokens": 8192,
    "input_cost_per_token": 0,
    "output_cost_per_token": 0,
    "litellm_provider": "ollama"
  }
}

Or via .aider.conf.yml:

1
2

model: ollama_chat/qwen2.5-coder:32b
ollama-api-base: http://127.0.0.1:11434

Using Aider with Any OpenAI-Compatible Endpoint (LM Studio, llama.cpp Server)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Point at LM Studio
export OPENAI_API_BASE=http://localhost:1234/v1
export OPENAI_API_KEY=lm-studio   # any non-empty value works

aider --model openai/qwen2.5-coder-32b-instruct

# Point at llama.cpp server
export OPENAI_API_BASE=http://localhost:8080/v1
export OPENAI_API_KEY=no-key

aider --model openai/local

Which Models Work Best with Aider

Aider maintains a public LLM leaderboard at aider.chat/docs/leaderboards that measures actual code editing performance. Check this resource for up-to-date rankings.

Based on community experience and the leaderboard as of early 2026:

Aider works best with models that have strong instruction-following — the model must respond in a specific diff format. Smaller models under 7B frequently fail to adhere to the required output format, making them unreliable for Aider.

Aider Workflow Example

1
2
3
4
5
6
7
8

# Start Aider session
cd ~/my-project
aider --model ollama_chat/qwen2.5-coder:32b src/api.py src/models.py

# Inside the session:
# > Add input validation to the create_user endpoint
# > Write tests for the validation logic
# > Refactor the database connection to use a connection pool

Aider reads the specified files, sends them as context, and writes the changes directly — then stages them for git commit. Use aider --no-auto-commits to review changes before committing.

Quick Reference: Picking the Right Stack

By VRAM / Budget

Recommended Starting Point (Most Users)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# 1. Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# 2. Pull models
ollama pull qwen2.5-coder:7b      # chat
ollama pull qwen2.5-coder:1.5b    # autocomplete
ollama pull nomic-embed-text       # embeddings

# 3. Install Continue.dev in VS Code
# Search "Continue" in VS Code Extensions

# 4. Configure ~/.continue/config.yaml (see Part 6)

From there, upgrade the chat model as your hardware allows, and consider LM Studio if you want a GUI for model management, or llama.cpp directly for maximum performance tuning.

Sources

• 5 Open-Source Coding LLMs You Can Run Locally in 2026 — Labellerr
• Best Models for Coding Locally in 2026 — InsiderLLM
• Comparing Top 7 LLMs/Systems for Coding in 2025 — MarkTechPost
• Codestral 22B, Qwen 2.5 Coder, and DeepSeek V2 Coder: Which AI Coder? — Deepgram
• Best LLM for Coding — Vellum
• Using Ollama with Continue: A Developer’s Guide — Continue Docs
• VS Code — Ollama Docs
• An Entirely Open-Source AI Code Assistant — Ollama Blog
• LM Studio Developer Docs
• OpenAI Compatibility Endpoints — LM Studio Docs
• Use OpenAI’s Responses API with Local Models — LM Studio Blog
• Local LLM Hosting Complete 2025 Guide — Medium
• Quantize Llama Models with GGUF and llama.cpp — Towards Data Science
• AI Model Quantization 2025 Guide — Local AI Zone
• llama.cpp GitHub Repository
• How Much GPU VRAM Do You Need for 7B, 33B, or 70B Models? — DatabaseMart
• Full Local LLM Setup Guide: CPU vs GPU vs Apple Silicon — ML Journey
• VRAM for 70B Models — SitePoint
• Apple Silicon vs NVIDIA CUDA: AI Comparison 2025 — Scalastic
• DeepSeek-Coder-V2 GitHub
• DeepSeek-Coder-V2 arXiv Paper
• StarCoder2 and The Stack v2 — arXiv
• WizardCoder — arXiv ICLR 2024
• Connecting to LLMs — Aider Docs
• Ollama — Aider Docs
• OpenAI Compatible APIs — Aider Docs
• Aider LLM Leaderboards
• config.yaml Reference — Continue Docs
• Continue Autocomplete Setup — Continue Docs
• Ollama REST API — GitHub
• OpenAI Compatibility — Ollama Blog
• CPU-Only LLM Inference Benchmarks — Challenging GPU Dominance, arXiv
• LLM Inference Speed Comparison — Ajit Singh
• I installed, quantized, and benchmarked top open-source Code LLMs — Medium