Local LLM Deep Dive: Ollama, Quantization, and Running AI on Your Own Hardware

The frontier AI services — Claude, GPT-4o, Gemini — are genuinely impressive. They are also cloud-bound, token-metered, and dependent on an internet connection. For developers and homelabbers who care about privacy, cost at scale, or just the satisfaction of owning the full stack, local LLMs have reached a point where they are genuinely useful for day-to-day coding work. This post covers everything you need to go from zero to a fully operational local AI setup: hardware selection, quantization formats, Ollama installation and configuration, model selection for coding tasks, and building a practical coding assistant that rivals cloud tools for many common tasks.

Why Run LLMs Locally?

Before diving into the technical details, it is worth being clear-eyed about the trade-offs. Local LLMs are not a free lunch, but for the right use cases they are the better choice.

The Case For

Privacy is absolute. When you run a model locally, your code, your prompts, and your data never leave your machine. For proprietary codebases, client data, or anything subject to NDAs or compliance requirements, this is often non-negotiable. Many enterprise developers have watched their companies ban cloud AI tools outright due to data leakage concerns. A local model sidesteps the entire problem.

Cost scales differently. Cloud AI APIs bill per token. At low volumes this is cheap. At high volumes — automated code review pipelines, IDE autocomplete firing hundreds of times per day, batch processing tasks — the economics flip. A one-time hardware investment (or a machine you already own) amortizes to essentially zero per-token cost.

No network round trip. For interactive use cases, a fast local model generating 40–80 tokens/sec on your own machine can feel snappier than a frontier API call that includes DNS resolution, TLS handshake, queuing, and the return trip over the wire. For IDE autocomplete especially, low latency matters more than raw quality.

Offline capability. Trains, planes, remote sites, network outages. A local model works everywhere, always.

Customization. You control the system prompt, the temperature, the context length, and the content filters (or absence thereof). You can fine-tune on your own code. You can create specialized model variants with persistent behaviors that cloud providers do not expose.

The Honest Trade-offs

Hardware cost. A capable GPU costs real money. If you already have a gaming PC with a modern NVIDIA card, the marginal cost is near zero. If you are buying dedicated hardware, budget $300–$1500+ depending on what you want to run.

Setup complexity. This is improving rapidly, but you will still encounter driver issues, VRAM constraints, and configuration tuning. This post is designed to minimize that friction.

Quality gap vs. frontier models. This is the honest one. As of early 2026, Claude 3.7 Sonnet and GPT-4o are still noticeably better than the best open-source models for complex multi-step reasoning. For straightforward coding tasks — function completion, refactoring, documentation, debugging familiar errors — the gap is much smaller. For complex architecture decisions or debugging novel multi-system failures, frontier models still have an edge. Know which tasks you are optimizing for.

Understanding the Hardware Landscape

The single most important concept in running local LLMs is: VRAM is the bottleneck, not compute speed.

When a model runs inference, it loads all of its weights into memory and keeps them there for the duration of the session. A 7 billion parameter model needs approximately 7 GB of memory just to hold the weights (more on exact calculations below). If those weights do not fit in VRAM, the model spills to system RAM or disk, and performance degrades by 10–50x.

The VRAM Math

The calculation is straightforward:

VRAM required ≈ parameters × bytes_per_weight + KV cache overhead

Weight size depends on the precision (quantization level):

Add 1–2 GB for the KV cache at moderate context lengths. So a 7B model at Q4_K_M quantization needs roughly 4–5 GB of VRAM, comfortably fitting a 6GB card with room for the KV cache.

Running on CPU

You can run models entirely on CPU with no GPU at all. The catch is speed. Where a capable GPU generates 30–80 tokens/second, a modern CPU typically manages 2–10 tokens/second for the same model. That is bearable for occasional queries but painful for interactive autocomplete.

If you have some VRAM but not enough for the full model, Ollama and llama.cpp support partial GPU offloading — as many layers as fit go on the GPU, the rest stay in RAM. Even partial offloading dramatically improves speed over CPU-only inference.

Consumer GPUs

NVIDIA RTX 3060 12GB (~$250 used): An outstanding value for local AI. The 12GB VRAM variant (there is also a 8GB variant — avoid it for AI) comfortably handles 7B models at full precision and 13B models at Q4. This is the sweet spot for most developers.

NVIDIA RTX 4070 Ti / 4070 Super (12–16GB): A significant step up in compute throughput over the 3060 while maintaining reasonable VRAM. 13B models run smoothly, 34B quantized models are feasible.

NVIDIA RTX 3090 / 4090 (24GB): With 24GB VRAM you can run 34B models at Q4 comfortably and push into 70B territory with aggressive quantization. The 4090 is the fastest consumer GPU for AI inference and a popular choice for power users who want to run large models without going to datacenter hardware.

AMD RX 7900 XTX (24GB): AMD’s ROCm software stack has improved substantially, and Ollama supports ROCm out of the box on Linux. Performance is within striking distance of comparable NVIDIA hardware, and the 24GB VRAM is attractive at the price point. Expect slightly more setup friction than NVIDIA.

Apple Silicon

Apple Silicon deserves its own section (covered in depth below), but the headline is: Apple’s unified memory architecture means RAM and GPU memory are the same pool. An M2 Max with 64GB of RAM has an effective “VRAM” of 64GB. This makes Apple Silicon uniquely capable of running large models on consumer hardware. The trade-off is bandwidth — even the M4 Max at ~400 GB/s memory bandwidth is slower than an RTX 4090 at ~1000 GB/s.

Server and Datacenter GPUs

NVIDIA A10 24GB (~$1500–$2000 used): A workstation/datacenter card that performs similarly to an RTX 3090 for inference but is built for 24/7 operation. Popular in homelab AI builds.

NVIDIA A100 40GB / 80GB: Professional datacenter hardware. The 80GB variant can run 70B models at full precision. Expensive and power-hungry but impressive. Used A100 40GB cards can be found for $5,000–$8,000.

NVIDIA H100: The current generation datacenter accelerator. Not practically relevant for home use, but included here for context when comparing against hosted inference costs.

Multi-GPU

Ollama and llama.cpp both support multi-GPU inference through tensor parallelism, splitting model layers across multiple cards. Two RTX 3090s giving you 48GB of effective VRAM can run 70B models at Q4 in full GPU mode. The speedup is roughly linear — two GPUs generate roughly 2x the tokens/second of one. Setup requires matching GPU models for best results.

Ollama: The Easiest Way to Run Local LLMs

Ollama is a runtime, model manager, and OpenAI-compatible API server bundled into a single binary. It abstracts away the complexity of llama.cpp, handles model downloads, and exposes a clean HTTP API and CLI. For most people, Ollama is the right starting point.

Installation

Linux:

1

curl -fsSL https://ollama.com/install.sh | sh

This installs the ollama binary, sets up a systemd service, and configures NVIDIA or AMD GPU support automatically if drivers are present.

macOS (Homebrew):

1
2

brew install ollama
# Or download the .app from https://ollama.com

Windows: Download the installer from https://ollama.com. Runs as a system tray application.

Docker:

1
2
3
4
5

# CPU only
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# NVIDIA GPU
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Core Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Download a model
ollama pull llama3.2
ollama pull qwen2.5-coder:14b
ollama pull codellama:13b

# Interactive chat session
ollama run llama3.2
ollama run qwen2.5-coder:14b

# List downloaded models
ollama list

# Show model details (parameters, context length, quantization)
ollama show qwen2.5-coder:14b

# Remove a model
ollama rm codellama:13b

# Start the API server manually (also runs on install)
ollama serve

# Run with verbose output to see tokens/sec
ollama run llama3.2 --verbose

The Ollama Model Library

Browse available models at https://ollama.com/library. Models are specified as name:tag where the tag indicates size and quantization. Some useful tag conventions:

1
2
3
4
5

ollama pull qwen2.5-coder          # default (usually 7B Q4_K_M)
ollama pull qwen2.5-coder:7b       # explicit 7B
ollama pull qwen2.5-coder:14b      # 14B parameter model
ollama pull qwen2.5-coder:14b-instruct-q4_K_M  # explicit quantization
ollama pull qwen2.5-coder:32b-instruct-q8_0    # 32B at Q8 precision

Modelfiles: Customizing Models

Ollama’s Modelfile format lets you create custom model variants with persistent system prompts, custom parameters, and modified defaults. This is powerful for creating specialized assistants.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Save as Modelfile
FROM qwen2.5-coder:14b

# Set a persistent system prompt
SYSTEM """
You are a senior Python developer working on a production FastAPI codebase.
When reviewing code:
- Prioritize type safety and Pydantic models
- Flag missing error handling
- Suggest async patterns where appropriate
- Use concise, actionable feedback
Never include unnecessary explanations. Get to the point.
"""

# Tune parameters
PARAMETER temperature 0.1        # Lower = more deterministic (better for code)
PARAMETER num_ctx 8192           # Context window (tokens)
PARAMETER num_gpu 99             # Offload all possible layers to GPU
PARAMETER top_p 0.9

1
2
3
4
5

# Build the custom model
ollama create python-expert -f Modelfile

# Use it
ollama run python-expert

REST API: OpenAI-Compatible Endpoint

Ollama exposes a REST API at http://localhost:11434. The /v1/chat/completions endpoint is OpenAI-compatible, meaning any tool or library that supports OpenAI’s API can point at Ollama instead.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Test the API directly
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:14b",
    "messages": [
      {"role": "system", "content": "You are a helpful coding assistant."},
      {"role": "user", "content": "Write a Python function to parse a JWT without a library."}
    ],
    "stream": false
  }'

# Ollama-native API (slightly simpler)
curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5-coder:14b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Key Environment Variables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Bind to all interfaces (to serve your homelab network)
export OLLAMA_HOST=0.0.0.0:11434

# Custom model storage location
export OLLAMA_MODELS=/mnt/fast-nvme/ollama-models

# Allow multiple models loaded simultaneously
export OLLAMA_MAX_LOADED_MODELS=3

# Handle parallel requests (useful for multi-user setups)
export OLLAMA_NUM_PARALLEL=2

# Keep model in memory longer (default: 5 minutes of inactivity)
export OLLAMA_KEEP_ALIVE=30m

On Linux with systemd, set these in /etc/systemd/system/ollama.service.d/override.conf:

1
2
3
4

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_KEEP_ALIVE=30m"

Then reload: sudo systemctl daemon-reload && sudo systemctl restart ollama

Quantization Explained

Quantization is the practice of reducing the numerical precision of a model’s weights. A model trained at float32 precision (4 bytes per weight) can be quantized to int4 (0.5 bytes per weight) — an 8x reduction in memory footprint with a surprisingly small quality loss when done well.

GGUF: The Standard Format

GGUF (GPT-Generated Unified Format) is the file format used by llama.cpp and Ollama for quantized models. It bundles the model weights, tokenizer, and metadata into a single file. When Ollama pulls a model, it is downloading a GGUF file.

GGUF filenames encode the model and quantization level:

Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
│               │   │        └── quantization method
│               │   └────────── precision level
│               └──────────────── parameter count
└──────────────────────────────── model family + variant

Quantization Levels

The K suffix in names like Q4_K_M indicates K-quants, a more sophisticated quantization scheme that applies different precision to different layers. The M suffix means “medium” block size. K-quants almost always outperform their non-K equivalents at the same file size.

Perplexity: Measuring Quality Loss

Perplexity is the standard academic measure of language model quality — lower is better. Here is a representative comparison for Llama 3.1 8B:

The practical takeaway: Q4_K_M loses about 2.7% perplexity versus full precision — imperceptible in most real-world tasks. Q3_K_M and below start showing up in output quality on complex reasoning tasks. For coding work, Q4_K_M is the standard choice, and Q8_0 when you can afford it.

Alternative Quantization Formats

For pure GPU inference (no CPU involved), alternative formats offer better speed and quality trade-offs:

GPTQ: An early GPU quantization format, widely available on HuggingFace. Supports 4-bit and 8-bit. Slightly lower quality than GGUF K-quants but runs fast on NVIDIA GPUs.

AWQ (Activation-aware Weight Quantization): A newer technique that identifies and protects the most important weights during quantization. Generally better quality than GPTQ at the same bit width.

ExLlamaV2: A highly optimized CUDA inference engine that supports its own quantization format (EXL2). Outstanding performance on NVIDIA GPUs, especially for speculative decoding. More complex to set up than Ollama but faster for power users.

For most people using Ollama, GGUF Q4_K_M is the practical standard. If you are building a high-throughput serving system on a dedicated NVIDIA GPU, investigate ExLlamaV2 or vLLM with AWQ.

Model Comparison for Coding Tasks

Not all models are created equal for code. The qualities that make a good coding model include: strong code completion, accurate debugging suggestions, clear explanation of unfamiliar code, ability to handle multi-file context, and good instruction following for refactoring tasks.

Models to Know

Qwen2.5-Coder (Alibaba): The current leader in open-source coding models across every size tier. Available in 0.5B, 1.5B, 3B, 7B, 14B, and 32B variants. Excellent at Python, JavaScript, Rust, Go, SQL, and shell scripting. Strong at code explanation, generation, and debugging. The 7B variant is a remarkable achievement in capability per parameter.

DeepSeek-Coder-V2 (DeepSeek): Strong performance on complex reasoning combined with code, particularly useful when problems require working through logic before writing code. The Lite variant (16B parameters, runs as an effective 2.4B active thanks to its MoE architecture) is extremely capable per resource unit.

Codestral (Mistral): Mistral’s dedicated coding model. Strong on Python, JavaScript, and SQL. The fill-in-the-middle capability makes it particularly good for autocomplete. Available through Ollama.

Llama 3.1/3.2 (Meta): General-purpose models that are genuinely good at code. The 3.1 8B model punches well above its weight class on coding benchmarks. The 3.2 3B model is surprisingly capable for its size. Best used when you want strong natural language + code in a single model.

Phi-3.5 Mini (Microsoft, 3.8B): Arguably the most impressive small model for coding tasks. Fits in 4GB of VRAM at Q4. Good for CPU-only or VRAM-constrained setups where you still want useful code assistance.

StarCoder2 (BigCode): A focused code completion model trained on a large corpus of permissively licensed code. Available in 3B, 7B, and 15B. Particularly strong at fill-in-the-middle for autocomplete use cases.

Comparison Table

*DeepSeek-Coder-V2 Lite uses MoE (Mixture of Experts) architecture — 16B total parameters, ~2.4B active per forward pass.

Model Selection by Hardware

4–6 GB VRAM (RTX 3060 8GB, GTX 1070 8GB, etc.):

• Primary: qwen2.5-coder:7b (Q4_K_M, ~4.5 GB)
• Autocomplete: phi3.5:3.8b or qwen2.5-coder:1.5b
• General: llama3.2:3b

8–12 GB VRAM (RTX 3060 12GB, RTX 3080 10GB, RTX 4070):

• Primary: qwen2.5-coder:14b (Q4_K_M, ~9 GB)
• Autocomplete: qwen2.5-coder:1.5b (leave VRAM for the main model)
• General: llama3.1:8b

16–24 GB VRAM (RTX 3090, RTX 4090, RTX 4080):

• Primary: qwen2.5-coder:32b (Q4_K_M, ~20 GB) or deepseek-coder-v2:16b
• Good runner-up: codestral:22b
• If you want quality over size: qwen2.5-coder:14b at Q8_0

24 GB+ (dual GPU, high-end Apple Silicon, datacenter cards):

• qwen2.5-coder:32b at Q8_0
• llama3.1:70b at Q4_K_M (~41 GB — needs dual 24GB or Apple Silicon 48GB+)

Setting Up a Local Coding Assistant

Having Ollama running is the foundation. The next step is connecting it to your actual development workflow.

Continue.dev: AI in Your IDE

Continue.dev is an open-source VS Code and JetBrains extension that connects to Ollama (and other backends) to provide in-editor chat and tab autocomplete. It is the most seamless way to integrate local LLMs into your development workflow.

Install from the VS Code marketplace: search for “Continue”.

Configure it by editing ~/.continue/config.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

{
  "models": [
    {
      "title": "Qwen2.5-Coder 14B (Chat)",
      "provider": "ollama",
      "model": "qwen2.5-coder:14b",
      "apiBase": "http://localhost:11434"
    },
    {
      "title": "Llama 3.1 8B (Fast)",
      "provider": "ollama",
      "model": "llama3.1:8b",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5-Coder 1.5B (Autocomplete)",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b",
    "apiBase": "http://localhost:11434"
  },
  "tabAutocompleteOptions": {
    "useCopyBuffer": false,
    "maxPromptTokens": 1024,
    "debounceDelay": 350
  },
  "contextProviders": [
    {"name": "code", "params": {}},
    {"name": "docs", "params": {}},
    {"name": "diff", "params": {}},
    {"name": "terminal", "params": {}},
    {"name": "problems", "params": {}},
    {"name": "folder", "params": {}}
  ],
  "slashCommands": [
    {"name": "edit", "description": "Edit selected code"},
    {"name": "comment", "description": "Write comments for code"},
    {"name": "share", "description": "Export conversation"},
    {"name": "cmd", "description": "Generate shell command"}
  ]
}

The key pattern here: use a fast small model (1.5B–3B) for tab autocomplete where low latency matters, and a larger capable model (14B+) for chat where quality matters. The two run simultaneously in Ollama without conflict.

To pull the autocomplete model:

1

ollama pull qwen2.5-coder:1.5b

Aider: AI Pair Programming in the Terminal

Aider is a command-line AI coding tool that works directly with your git repository. It understands your project structure, edits files directly, and commits changes. Think of it as a terminal-native coding agent backed by your local Ollama.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

pip install aider-chat

# Point Aider at your local Ollama
aider --model ollama/qwen2.5-coder:14b

# Or set it via environment variable
export OLLAMA_API_BASE=http://localhost:11434
aider --model ollama/qwen2.5-coder:14b --no-auto-commits

# Common workflows
aider --model ollama/qwen2.5-coder:14b src/api/routes.py src/models/user.py
# Then type your instruction in the prompt:
# > Add input validation to all POST endpoints using Pydantic

Aider maps your repository, reads relevant files into context, makes targeted edits, and optionally commits with descriptive messages. On a 14B model you will see 15–30 tokens/second, which is fast enough for fluid interactive use.

OpenWebUI: Self-Hosted ChatGPT Interface

OpenWebUI is a polished, self-hosted web interface for Ollama that provides a ChatGPT-like experience with model management, chat history, RAG document upload, and multi-user support.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# docker-compose.yml
services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    ports:
      - "3000:8080"
    volumes:
      - openwebui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change-this-to-a-random-string
    depends_on:
      - ollama
    restart: unless-stopped

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  openwebui_data:
  ollama_models:

1
2

docker compose up -d
# Access at http://localhost:3000

OpenWebUI’s RAG feature lets you upload documents (PDFs, code files, text) and query your Ollama models against them. This is excellent for asking questions about internal documentation, runbooks, or large codebases that exceed the model’s context window.

LM Studio: GUI for Model Testing

LM Studio provides a graphical interface for downloading models from HuggingFace, testing them interactively, and running an OpenAI-compatible local server. It is available for macOS, Windows, and Linux. For users who prefer a GUI for initial setup and model exploration, LM Studio is the quickest path to a working local model. Its HuggingFace integration makes it easy to browse and download GGUF models directly.

Apple Silicon Setup

Apple Silicon deserves dedicated attention because it represents a genuinely different architecture for running LLMs — one that in many scenarios outperforms consumer NVIDIA hardware for the use cases most developers care about.

Why Apple Silicon is Different

The key is unified memory architecture (UMA). On a standard PC, the CPU has system RAM and the GPU has its own dedicated VRAM — two separate pools with a PCI Express bus between them. Moving model weights to the GPU requires copying them across the PCIe bus.

On Apple Silicon, a single pool of high-bandwidth LPDDR5 RAM is shared between the CPU, GPU, and Neural Engine. There is no bus transfer overhead, and the GPU can use the full RAM capacity as its “VRAM.” An M3 Max with 128GB of unified memory effectively has 128GB of GPU memory.

The bandwidth comparison matters:

The M4 Max at 546 GB/s approaches the RTX 3090/4090 in bandwidth, which is the primary determinant of inference speed. The advantage is that you get that bandwidth with up to 128GB of addressable model memory, versus only 24GB on consumer NVIDIA cards.

Ollama on macOS

Ollama automatically uses Metal GPU acceleration on Apple Silicon. No additional configuration is needed.

1
2
3
4
5
6
7

# Install
brew install ollama

# Start the service
ollama serve

# Or use the .app from ollama.com — it runs in the menu bar

Verify GPU acceleration is active:

1. Open Activity Monitor
2. Go to Window → GPU History
3. Run ollama run llama3.1:8b and ask a complex question
4. GPU utilization should spike noticeably

Tokens/Second Benchmarks on Apple Silicon

These are representative benchmarks for Qwen2.5-Coder 14B Q4_K_M (a realistic daily-driver coding model):

For Llama 3.1 70B Q4_K_M (requires 40+ GB RAM, so M2 Max 64GB or better):

Recommended Models by Apple Silicon Config

M1/M2 8GB: VRAM-constrained. Stick to small models.

• llama3.2:3b (Q4_K_M, ~2 GB)
• qwen2.5-coder:1.5b (Q4_K_M, ~1 GB)
• phi3.5:3.8b (Q4_K_M, ~2.8 GB)

M1/M2 16GB: Viable for 7B models.

• qwen2.5-coder:7b (Q4_K_M, ~4.5 GB) — good coding assistant
• llama3.1:8b (Q4_K_M, ~5 GB)
• Leave 8–10 GB for the OS and other apps

M1/M2 Pro 32GB: The sweet spot for daily use.

• qwen2.5-coder:14b (Q4_K_M, ~9 GB)
• llama3.1:8b at Q8_0 for near-full quality

M2/M3 Max 32–64GB: Serious capability.

• qwen2.5-coder:32b (Q4_K_M, ~20 GB)
• llama3.1:70b (Q4_K_M, ~41 GB) on 64GB config
• deepseek-r1:32b for reasoning tasks

M4 Max 64–128GB: Flagship performance.

• llama3.1:70b at Q8_0 (~70 GB) — near-full quality 70B
• deepseek-r1:70b (Q4_K_M, ~41 GB)
• Multiple large models loaded simultaneously

llama.cpp Directly for Maximum Performance

Ollama uses llama.cpp under the hood, but you can run llama.cpp directly for more control and marginally better performance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Install via Homebrew
brew install llama.cpp

# Download a GGUF directly from HuggingFace
# Example: Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf

# Run inference
llama-cli \
  -m /path/to/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf \
  -n 512 \
  --temp 0.1 \
  -p "Write a Python function to parse an IPv6 address" \
  --gpu-layers 35

# Run as a server (OpenAI-compatible)
llama-server \
  -m /path/to/model.gguf \
  --port 8080 \
  --gpu-layers 35 \
  --ctx-size 8192

The --gpu-layers flag controls how many transformer layers are offloaded to the Metal GPU. Set it to 999 or -1 to offload as many as possible. Monitor memory pressure in Activity Monitor and reduce if the system starts swapping.

NVIDIA GPU Setup on Linux

Driver Installation and Verification

Before Ollama can use your GPU, you need working NVIDIA drivers and CUDA:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Check if drivers are already installed
nvidia-smi

# Install drivers on Ubuntu/Debian
sudo apt update
sudo apt install nvidia-driver-545  # or the latest stable version
sudo reboot

# Verify after reboot
nvidia-smi
# Should show GPU name, driver version, CUDA version, and memory usage

# Install CUDA toolkit (optional, Ollama does not require it but other tools do)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-toolkit-12-3

Ollama with NVIDIA GPU

Ollama detects NVIDIA GPUs automatically when drivers are installed. Verify GPU usage:

1
2
3
4
5
6

# Start Ollama and run a model
ollama run qwen2.5-coder:14b "Explain what a monad is"

# In another terminal, watch GPU utilization
nvidia-smi -l 1   # refresh every second
# VRAM usage should show the model loaded and utilization should be high during generation

Docker with GPU Passthrough

For Docker-based setups, install the NVIDIA container toolkit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Install nvidia-container-toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Test GPU passthrough
docker run --gpus all nvidia/cuda:12.3-base-ubuntu22.04 nvidia-smi

# Run Ollama with GPU
docker run -d \
  --gpus all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

GPU Monitoring Tools

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# nvidia-smi: built-in, shows VRAM usage and utilization
nvidia-smi
nvidia-smi dmon -s u   # continuous monitoring of utilization
nvidia-smi dmon -s m   # continuous monitoring of memory

# nvtop: htop-style GPU monitor (recommended)
sudo apt install nvtop
nvtop

# gpustat: compact GPU status (good for scripts)
pip install gpustat
gpustat -i 1   # refresh every second
watch -n 1 gpustat

Managing VRAM

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# See what models are currently loaded in Ollama
curl http://localhost:11434/api/ps

# Unload a specific model from VRAM
curl http://localhost:11434/api/generate \
  -d '{"model": "llama3.1:8b", "keep_alive": 0}'

# Set global keep-alive: model unloads after 5 minutes idle
export OLLAMA_KEEP_ALIVE=5m

# Limit loaded models (prevents VRAM exhaustion)
export OLLAMA_MAX_LOADED_MODELS=1

Performance Tuning

Context Window

The num_ctx parameter sets the context window size — how many tokens the model can “see” at once. Larger context means more VRAM and more compute per token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Default is usually 2048 or 4096 tokens
# Increase for long conversations or large file analysis

# Via Modelfile
PARAMETER num_ctx 16384

# Via API
curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5-coder:14b",
    "options": {"num_ctx": 16384},
    "messages": [{"role": "user", "content": "Review this entire file..."}]
  }'

Rule of thumb: 4096 tokens ≈ ~3000 words ≈ ~100–150 lines of code. For reviewing full files, set num_ctx to 8192 or 16384. Be aware this roughly doubles VRAM consumption for the KV cache.

Temperature and Sampling

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# For code generation: low temperature (0.0-0.2) for determinism
PARAMETER temperature 0.1

# For creative brainstorming: higher temperature (0.7-0.9)
PARAMETER temperature 0.7

# top_p: nucleus sampling — keep tokens comprising top 90% of probability mass
PARAMETER top_p 0.9

# top_k: only consider top K tokens at each step
PARAMETER top_k 40

For coding tasks, temperature 0.1 is almost always the right setting. High temperature introduces randomness which is great for creative writing and catastrophic for code that needs to compile.

Benchmarking Your Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Run with verbose to see generation speed
ollama run qwen2.5-coder:14b --verbose "Write a merge sort in Python"

# Output includes:
# total duration:   12.4s
# load duration:    1.2s
# prompt eval count: 23 tokens
# prompt eval rate: 450 tokens/s
# eval count:       312 tokens
# eval rate:        28.7 tokens/s   <-- this is the key number

# For systematic benchmarking, use llama-bench (from llama.cpp)
llama-bench -m /path/to/model.gguf -p 512 -n 128

Target performance numbers for a good daily-driver experience:

• Autocomplete (1.5B model): 60+ tokens/sec
• Chat (7B model): 30+ tokens/sec
• Chat (14B model): 15+ tokens/sec
• Chat (70B model): 5+ tokens/sec (acceptable, not fast)

Privacy and Network Setup

Keeping Ollama Strictly Local

By default, Ollama listens on 127.0.0.1:11434 — local only. This is the safest configuration. Do not change this unless you specifically need network access.

Exposing to Your Homelab Network

If you want to access Ollama from other machines on your LAN:

1
2
3

# In /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Warning: This exposes the Ollama API to your entire local network with no authentication. Anyone on your LAN can query your models and potentially exfiltrate conversation history. Only do this on a trusted network.

Reverse Proxy with Authentication

For any network-accessible Ollama, add authentication via a reverse proxy. Using Traefik with basic auth:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# traefik dynamic config: ollama-auth.yml
http:
  routers:
    ollama:
      rule: "Host(`ollama.homelab.local`)"
      service: ollama
      middlewares:
        - ollama-auth
      tls: {}

  middlewares:
    ollama-auth:
      basicAuth:
        users:
          - "admin:$2y$10$..."   # htpasswd generated hash

  services:
    ollama:
      loadBalancer:
        servers:
          - url: "http://localhost:11434"

Generate a password hash: htpasswd -nB admin

Tailscale for Remote Access

The cleanest solution for accessing your home Ollama from anywhere (coffee shop, office, travel) is Tailscale. Your machines get a private Tailscale IP (100.x.x.x) that is accessible only to your Tailscale account, encrypted end-to-end, with no port forwarding required.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# Once connected, access Ollama by Tailscale IP from any device on your tailnet
# http://100.x.x.x:11434

# Or use Tailscale MagicDNS
# http://my-ai-server:11434

# Set Ollama to listen on Tailscale interface only (more secure than 0.0.0.0)
export OLLAMA_HOST=100.x.x.x:11434  # your Tailscale IP

With Tailscale, you can configure Continue.dev on your laptop to point at your home Ollama server:

1
2
3
4
5
6
7
8

{
  "models": [{
    "title": "Home Server - Qwen2.5-Coder 32B",
    "provider": "ollama",
    "model": "qwen2.5-coder:32b",
    "apiBase": "http://my-homelab-server:11434"
  }]
}

Practical Use Cases and Prompting Tips

Effective Prompting Patterns for Code

The system prompt is your most powerful lever. Unlike cloud providers, local Ollama gives you full control over it. Tailor it to your stack:

You are a senior Go developer. When reviewing code:
- Flag any goroutine leaks or missing context cancellation
- Point out missing error wrapping (use fmt.Errorf with %w)
- Identify places where defer could prevent resource leaks
- Note any anti-patterns specific to the standard library
Respond in bullet points. Be direct and concise.

Common Use Cases

Code review: Paste a function and ask for feedback.

Review this function for bugs, edge cases, and Go idioms.
Do not rewrite it — give me a bulleted list of specific issues.

[paste code]

Debugging from error output:

I'm getting this error in production. Here is the traceback:
[paste traceback]

Here is the relevant code:
[paste code]

What is causing this, and what is the minimal fix?

Generating documentation:

Write a GoDoc comment block for this function.
Include parameter descriptions, return values, and one usage example.

[paste function signature and body]

Writing commit messages:

Write a conventional commit message for this git diff.
Use the format: type(scope): description
Follow it with a brief body explaining the why, not the what.

[paste git diff]

Explaining unfamiliar code:

Explain this code to me as if I have never seen this library before.
Focus on what problem it solves, not line-by-line description.

[paste unfamiliar code]

Refactoring with constraints:

Refactor this function to:
1. Extract the validation logic into a separate helper
2. Replace the nested if-else chain with early returns
3. Add a unit test for the edge cases

Do not change the function signature or behavior.
[paste code]

Sizing Your Context Request

Local models have hard context limits. Be deliberate about what you include:

• For single-function review: 4096 tokens is plenty
• For file-level review: use 8192–16384 tokens, set num_ctx accordingly
• For multi-file refactoring: either use Continue.dev’s @folder context, use Aider (which manages context automatically), or chunk the task manually
• For entire codebases: use RAG (OpenWebUI’s document upload, or a tool like llm with embeddings) rather than trying to fit everything in the context window

Putting It All Together

Here is a practical recommended setup for a developer who wants a local coding assistant:

Hardware: An existing machine with an RTX 3060 12GB, or a Mac with 32GB+ unified memory. Either handles the recommended models comfortably.

Models to pull:

1
2
3

ollama pull qwen2.5-coder:14b       # primary chat/review model
ollama pull qwen2.5-coder:1.5b      # fast autocomplete
ollama pull llama3.1:8b             # general-purpose fallback

IDE integration: Continue.dev with qwen2.5-coder:1.5b for autocomplete and qwen2.5-coder:14b for chat.

Terminal workflow: Aider for making multi-file changes directly in your git repo.

Web interface: OpenWebUI via Docker for longer conversations, document-based RAG, and experimenting with different models.

Remote access: Tailscale so your home GPU is available from your laptop at the coffee shop.

The result is a private, offline-capable AI coding assistant that costs nothing per query, responds in under a second for autocomplete, and handles most common coding tasks — code review, documentation, debugging, and refactoring — without sending a single character of your code to an external server.

For complex multi-step architecture work or unfamiliar problems where model quality matters most, frontier cloud models still have the edge. The practical answer is not “local or cloud” but “local for high-volume routine tasks, cloud for complex high-stakes work.” That combination gives you the best of both worlds: cost efficiency and privacy where they matter, maximum capability where you need it.