Jan: The Open-Source Local LLM Runner Coming for Ollama and LM Studio

For two years the local-LLM conversation came down to a fork in the road: Ollama if you live in a terminal and want a daemon you can script, LM Studio if you want a polished desktop app to click around in. Both work. Both have a catch. Ollama’s UX is thin and its model packaging is its own walled garden; LM Studio is closed-source and you are trusting a binary blob with your prompts. Jan is the project that looked at both and asked why you should have to choose between an open license, a real GUI, and a server you can point your code at.

Jan is an open-source (Apache 2.0) local AI platform that ships a ChatGPT-style chat interface, a hardware-aware model hub, an extension system, and an OpenAI-compatible API server — all in one application, all running on your machine by default. Under the hood it is powered by Cortex, the team’s C++ inference engine, which runs on llama.cpp out of the box and can swap in ONNX or TensorRT-LLM. This post is a practical evaluation: what Jan actually is, where it genuinely beats the incumbents, where it doesn’t, and how to tune the engine when the defaults leave performance on the table.

What Jan Actually Is

It helps to be precise, because “local LLM app” covers three different jobs that Jan tries to do at once:

1. A chat client — a desktop GUI where you talk to models, manage threads, attach files and images, and wire up tools. This is the LM Studio comparison.
2. A model runner — the thing that downloads GGUF weights and executes them on your CPU/GPU. This is the Ollama and llama.cpp comparison.
3. A local API server — an OpenAI-compatible endpoint at http://localhost:1337/v1 that your own scripts and apps can call as a drop-in replacement for api.openai.com. This is the part that turns Jan from a toy into infrastructure.

The engine doing the real work is Cortex (cortex.cpp), a standalone C++ runtime the Jan team built explicitly as an Ollama alternative. By default Cortex drives llama.cpp — the same Georgi Gerganov / ggml-org library that Ollama and LM Studio also wrap — so raw token throughput is in the same ballpark as those tools on identical hardware and quantization. Cortex is multi-engine, though: it can also load ONNX Runtime models and TensorRT-LLM builds, which matters if you have NVIDIA hardware and want to trade portability for speed.

Everything is local-first. Models, threads, settings, and extensions live on disk. There is no account, no required login, and no telemetry on by default. Because it is Apache 2.0, you can read exactly what it does, fork it, or ship it inside a product without licensing drama — which is the single biggest structural difference from LM Studio.

The Three-Way Comparison

The honest framing is that these tools overlap but optimize for different operators. Here is how they line up in mid-2026:

A few of these deserve unpacking rather than a checkmark.

Model distribution. Ollama’s ollama pull llama4 pulls from Ollama’s own registry with its own manifest format. Jan and LM Studio pull GGUF files straight from Hugging Face, which means you get the entire community quant ecosystem (Bartowski, unsloth, mradermacher) without waiting for a maintainer to repackage. If you want a specific imatrix quant of a niche fine-tune, Jan reaches it; Ollama often doesn’t have it.

The server story. All three expose an OpenAI-compatible API, and this is where the “which one” question often dissolves — your application code doesn’t care, it just points base_url at a different port. Ollama still wins for pure headless server deployments because it was designed daemon-first. Jan’s server is real and improving, but the app is GUI-centric; running it truly headless on a server is more work than systemctl enable ollama.

Installing and Running

Jan ships native installers for macOS, Windows, and Linux. On Linux you grab the AppImage or .deb from the releases page. The first run is deliberately boring: open the app, go to the Hub, pick a model, download, chat.

The interesting part is the model hub, which is genuinely better than the competition at one specific thing: telling you whether a model will actually run before you spend 40 GB of bandwidth on it. Jan reads your hardware and shows a colored fit pill on every quantization:

Qwen3.5-27B
  ├─ Small     (Q4_K_M, 16.4 GB)   [ Fits ]          ← Recommended
  ├─ Balanced  (Q5_K_M, 19.2 GB)   [ May be slow ]
  └─ Large     (Q8_0,   28.9 GB)   [ Won't fit ]

A GPU backend is only chosen when a card reports at least 6 GB of VRAM; below that Jan falls back to CPU automatically. For multimodal models the VRAM precheck also accounts for the sibling mmproj.gguf projector file, which is a common cause of out-of-memory crashes in other tools that only weigh the main weights.

For the server, you don’t need the GUI to be the focus — enable the local API server from settings (or via the jan CLI) and it listens on http://localhost:1337. From there it is a standard OpenAI call:

1
2
3
4
5
6
7

curl http://localhost:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-27b",
    "messages": [{"role": "user", "content": "Explain RoPE scaling in two sentences."}],
    "temperature": 0.6
  }'

Point any OpenAI SDK at that base URL and your existing code works unchanged. This is the same pattern you would use to put Jan behind a router like LiteLLM when you want to mix local and hosted models behind one endpoint.

The v0.8 Engine Overhaul

Two recent releases meaningfully changed how Jan runs models, and they are worth knowing because they fix the two things that historically made local runners annoying.

Jan v0.8.0 replaced the old “one server process per loaded model” design with a single llama.cpp router process that loads and unloads models on demand. Practically, this means you can have several models configured and Jan manages which are resident in memory, rather than spawning a heavyweight server for each. The same release added multi-token prediction (speculative-style decoding for supported models) and inline MCP tool approval so you can authorize a tool call from the chat without digging through settings.

Jan v0.8.1 followed with an Anthropic-compatible custom provider type (so you can add Claude-style endpoints, not just OpenAI-style ones), per-message error surfacing instead of a silent failed turn, and a full overhaul of the llama.cpp settings UI — which is what makes the tuning section below actually usable.

If you are evaluating Jan against an older blog post or a six-month-old screenshot, assume it is out of date. The engine internals moved fast this spring.

Tuning the llama.cpp Engine

Out of the box the defaults are fine for most people. The reason to open the engine settings is one of three situations: a model won’t fit, a model fits but runs slowly, or you are serving multiple concurrent requests and want throughput. Jan exposes the relevant llama.cpp knobs without making you write a command line.

The engine-wide settings:

And the per-model overrides, which is where most of the real tuning happens:

• Context Size (ctx_size) — bigger context costs VRAM quadratically-ish via the KV cache. If you don’t need 128K, don’t allocate it.
• GPU Layers (n_gpu_layers) — how many transformer layers to offload to the GPU. The single most important dial for the “fits but slow” case: push as many layers onto the GPU as VRAM allows.
• Disable KV Offload (no_kv_offload) — keeps the KV cache in system RAM when VRAM is tight, trading speed for the ability to run at all.
• Batch Size, custom chat template, and sampling parameters.

For memory-constrained boxes, three flags matter most:

• MLock pins the model in RAM so the OS can’t swap it out — eliminates the periodic stalls you get when the kernel pages model weights to disk.
• Disable mmap is the fix when a model crashes on load or thrashes; it forces a full read into RAM instead of memory-mapping.
• KV Cache Quantization to q8_0 roughly halves the cache footprint with minimal quality loss, which is often the cheapest way to fit a longer context. q4_0 halves it again but you’ll start to feel it on long generations.

A worked example: you have a 24 GB RTX 4090 and want to run a 32B model at a long context. The model weights at Q4_K_M are ~19 GB. An f16 KV cache for 32K context might be another ~8 GB — you’re over budget and Jan will refuse or fall back to CPU. Drop the KV cache to q8_0, set n_gpu_layers to the maximum that fits, and you reclaim enough to run fully on-GPU. That single change is frequently the difference between 6 tokens/sec and 45.

The advanced section also exposes RoPE scaling (for stretching a model’s context beyond its trained length), Mirostat sampling, grammar/JSON-schema-constrained output, and a max-concurrent-models cap for the router (0 = unlimited, 1 = strict single model). For background on why these quantization choices behave the way they do, the quantization deep dive covers GPTQ, AWQ, and GGUF in detail.

Tools, MCP, and Agentic Use

This used to be Jan’s weak spot. The OpenAI-compatible API did not cleanly expose function-calling endpoints, so building tool-using agents on top of raw Jan meant prompt-engineering your own parser. That changed: Model Context Protocol support graduated to stable in v0.6.9 and now works without experimental flags, with inline approval added in v0.8.0.

The practical guidance is the same as anywhere else in the local-agent world: tool calling is a model capability, not just an app feature. A 3B model will hallucinate its way through a tool schema. Use models that were actually trained for function calling — Qwen3, Gemma 3/4 function-calling variants, and the larger instruct models. If you wire Jan into MCP servers, the model behind it has to be competent enough to use them, and on consumer VRAM that realistically means a 14B model or larger.

This is also the natural bridge to running a full agent on top of Jan’s server. Jan is a fine inference backend for an agent framework — point the agent’s OpenAI base URL at :1337 — but if you want a purpose-built agent runtime with memory and skills, see the companion post on the Hermes Agent desktop app, which can target a local endpoint the same way.

Where Jan Falls Short

No tool wins on every axis, and pretending otherwise helps nobody.

• It is younger and smaller than the incumbents. Fewer tutorials, fewer Stack Overflow answers, a smaller plugin ecosystem. When something breaks you are more often reading source than finding a forum thread.
• It is not Ollama for headless servers. If your goal is a daemon on a Linux box that ten services hit over the network, Ollama’s daemon-first design is still less friction. Jan can do it; it just wasn’t born for it.
• Throughput is llama.cpp-class, not vLLM-class. Because the default engine is llama.cpp, Jan is excellent for one-or-a-few concurrent users and not the right tool for serving a hundred. For that you want the production engines covered in the vLLM/SGLang/TensorRT-LLM shootout. Cortex’s TensorRT-LLM backend narrows the gap on NVIDIA hardware but adds build complexity.
• The desktop app is an Electron-style footprint. Heavier on RAM than Ollama’s lean daemon. If you only ever use the API, that overhead is pure cost.
• Tool calling, while improved, still trails the polish of hosted APIs. Local function-calling reliability is a model problem as much as a Jan problem, but it means agentic workflows need more babysitting than they would against a frontier cloud model.

For a broader survey of the local-inference landscape that Jan sits inside, the self-hosted AI inference and local LLMs with Ollama posts give the surrounding context.

The Verdict

Jan is the right default for a specific and increasingly common profile: you want a real graphical app to explore models in, you want an OpenAI-compatible server to build against, and you are not willing to give up an open license or send your prompts through a closed binary. It hits all three, and the v0.8 engine work closed most of the gap that used to make it feel like the third-place option.

Choose Ollama if your world is the terminal and you mostly want a scriptable daemon. Choose LM Studio if you don’t care about the license and want the most click-polished experience today. Choose Jan if you want to own the entire stack — UI, runner, and server — under Apache 2.0, with a model hub that won’t let you download something your hardware can’t run. For a lot of homelab and developer setups, that last description is exactly the one that fits, and the gap to the incumbents is now small enough that the open license is the deciding vote.

Sources

• Jan — Open-Source ChatGPT Replacement
• Jan Docs — Local AI Engine (llama.cpp)
• Jan Docs — Local API Server
• Jan Docs — Model Context Protocol (MCP)
• Jan v0.8.0: Multi-Token Prediction, llama.cpp Router Mode & Inline MCP Approval
• Jan v0.8.1: Anthropic-Compatible Custom Providers, Per-Message Errors & llama.cpp Settings Overhaul
• Jan v0.6.9: Image support, stable MCP, and powerful model tools
• Running Local LLMs in 2026: Ollama, LM Studio, and Jan Compared (DEV Community)
• Ollama vs LM Studio vs llama.cpp vs Jan.ai (runaihome)
• 7 Best LLM Tools To Run Models Locally — Unite.AI