Personal Knowledge Management with AI

I have used roughly twelve personal knowledge management systems over the past decade. I have tried Notion, Roam, Logseq, Bear, Apple Notes, Confluence (personal instance, which is a special kind of madness), a custom SQLite-backed tagging system I built myself, plain text files in a Dropbox folder, DEVONthink, Evernote before it became a cautionary tale, Zettelkasten on index cards for about three weeks, and finally Obsidian. Along the way I built elaborate hierarchies, defined taxonomies, wrote personal ontologies, designed templates for seventeen different note types, and — this is the part that takes a while to admit — produced almost no useful output from any of it.

The notes accumulated. The organization evolved. The retrieval stayed broken. The writing never happened.

What follows is not a guide for people who want to build a maximally sophisticated PKM system. There are plenty of those, and most of them will lead you exactly where I went. This is a guide for people who want to capture knowledge in a way that actually generates value — written from the perspective of someone who spent years confusing the map for the territory and eventually found a setup that is boring enough to use consistently and good enough to be worth using at all. AI is genuinely part of that setup now, but not in the way most PKM content suggests.

The Promise vs. The Reality

Why PKM Appeals to Intellectually Curious People

Personal knowledge management promises something intoxicating: that you will never lose a good idea, that the connections between disparate subjects you have studied will become visible, that you will be able to build on past work instead of rediscovering it, and that the accumulated knowledge of your reading life will compound like interest. For people who read widely, work across multiple domains, and feel perpetually behind on synthesizing what they know, this promise lands hard.

The framing is flattering. You are not just taking notes — you are building a second brain, an exocortex, a personal knowledge graph. The vocabulary of systems thinking gets applied to the activity of writing things down. It feels like serious intellectual infrastructure work.

The problem is that the promise is real but the path to it is much narrower than the PKM content ecosystem implies. The value in a knowledge management system comes almost entirely from retrieval and expression — from finding the right note at the right moment and turning it into something usable. It does not come from the sophistication of the system itself. A flat folder of text files with good search delivers more value than an intricate Zettelkasten with broken retrieval habits. The tooling matters far less than the practice, and the practice is much harder to sustain than any tutorial will tell you.

The Building-the-Library Trap

There is a specific failure mode that PKM content never names directly: spending time building the library instead of reading the books. You create the perfect folder structure before you have any notes to put in it. You design your tagging taxonomy before you have identified any recurring themes in your thinking. You build templates for six different note types before you have written twenty notes of any kind. You watch videos about Zettelkasten methodology instead of reading the papers you wanted to take notes on.

This is not laziness or lack of discipline. It is a cognitively easier version of the real work. Designing systems is concrete and completable — you can feel done after a good session of organizing. Actually reading, thinking, and writing is open-ended and uncomfortable. The system becomes a displacement activity for the harder task, and it is especially seductive because it masquerades as preparation.

The symptom is measurable: if you have spent more hours configuring your PKM than you have spent writing in it, you are in the trap.

What Most Notes Never Do

The uncomfortable statistical reality is that most notes, once captured, are never re-read. This has been observed anecdotally by essentially everyone who has maintained a PKM system for more than a year, and it makes intuitive sense once you think about it. Notes are often captured in a context that is no longer active — you were researching a technology you did not end up using, you were reading a book in a domain you moved away from, you were processing a meeting whose decisions have long since been superseded. The note was useful at capture time. It has no future value.

This does not mean capturing notes is useless. It means the expected value of any individual note is low, which means the cost of capturing and organizing each note needs to be proportionately low. Complex organization schemes impose a fixed cost per note regardless of whether that note will ever be retrieved. When the cost exceeds the expected return, the rational response is to stop taking notes, and that is exactly what happens when systems become too heavy.

What PKM Actually Enables When It Works

The genuine value of a PKM system, when it works, is narrower than the promise but real. It falls into roughly three categories.

First: returning to a topic you have studied before without starting from scratch. When you took notes during a deep dive on database indexing strategies six months ago, having those notes means you can pick up where you left off instead of re-reading the same articles. This is the most reliable source of value in any PKM system and it requires only basic capture and search.

Second: connecting ideas across time and domain in a way that generates original thinking. This is the zettelkasten ideal — the note you wrote about feedback loops in ecological systems linking to a note about organizational debt, producing an insight neither note contained independently. This does happen, but it requires enough density of atomic notes and enough active linking that it only pays off after years of consistent practice. Most people abandon the system before they reach that density.

Third: building reference material you actually use. A personal runbook, a curated collection of code snippets, a set of interview question notes, a library of prompts that work. This kind of reference-oriented PKM is underrated because it is unglamorous — it does not invoke the second brain mythology — but it delivers consistent, measurable value.

The Minimum Viable PKM

The 80/20 version of a PKM system is: a place to capture things quickly (daily note or inbox), a way to find things later (full-text search), and a habit of writing brief notes after doing any substantial research or project work. Everything else is optimization. Not waste, necessarily — but optimization that should be added only after the minimum is working.

The Second Brain Framework

CODE: Capture, Organize, Distill, Express

Tiago Forte’s Building a Second Brain introduced a workflow called CODE — Capture, Organize, Distill, Express — that became the dominant framing for modern PKM. It is worth understanding because it is genuinely better than what most people do, and also worth critiquing because the content ecosystem around it has become as elaborate as the Zettelkasten literature in ways that obscure its core simplicity.

Capture means saving anything potentially useful to an inbox before deciding what to do with it. The bar for capture is low by design — you are not deciding whether something is valuable at capture time, you are preserving the option to use it later. Forte recommends capturing to a central inbox from any device, any context, using whatever friction-free method is available.

Organize means moving captured items to a location based on where they will be useful, not what they are about. This is the PARA system, described below. The key insight is that actionability is a better organizing principle than topic.

Distill means progressively extracting the essence of a note through a process called progressive summarization — highlighting the most important passages, then bolding the most important of those, then writing a brief executive summary at the top. The goal is to make future-you able to understand the key point in thirty seconds without re-reading the whole note.

Express means turning your notes into outputs — writing, presentations, recommendations, code. Forte is explicit that expression is the point. Notes that are never expressed as output have delivered no value regardless of how well organized they are. This is the part most PKM content glosses over because it is also the hardest part.

The honest critique of CODE is that it requires sustained discipline across all four phases, and the phases have very different cognitive costs. Capture is easy. Organize is easy enough if PARA is working. Distill is moderately hard — progressive summarization takes time and judgment. Express is genuinely hard, and it requires not just discipline but something to say. You can have a perfect CODE system and still produce nothing if the express phase never happens, because the first three phases do not generate ideas, they only preserve and organize inputs. The ideas have to come from you.

PARA: Projects, Areas, Resources, Archives

PARA is the organizational structure that underlies the CODE methodology. Rather than organizing by topic — which leads to an ever-growing hierarchy of folders that becomes harder to navigate as it grows — PARA organizes by actionability.

Projects are things you are actively working toward a specific outcome. They have a defined completion state. A project folder for “Q3 infrastructure audit” or “migrate service to Kubernetes” or “write PKM blog post” contains everything relevant to that goal and will be archived when the project completes.

Areas are ongoing responsibilities with no end state. “Server administration,” “home lab,” “professional development,” “health.” These are domains you manage continuously rather than complete.

Resources are reference material for topics you are interested in. Notes and collected material on machine learning, networking fundamentals, woodworking, or any other topic that is not an active project or ongoing responsibility.

Archives are inactive items from the other three categories. Completed projects, former areas of responsibility, resources you no longer actively consult.

The practical advantage of PARA over topic-based organization is that the question “where should this go?” has a more constrained answer space. You are not choosing from among forty topic folders. You are asking: is this related to an active project? If yes, put it there. Is it related to an ongoing area of responsibility? Put it there. Is it general reference material? Put it in resources. The decision takes seconds instead of minutes, which matters enormously for building a consistent capture habit.

PARA also naturally surfaces what you are actually working on. Your Projects folder contains your current commitments. If there are twenty projects in it, that is useful information — you may be overcommitted. If a project folder has not been touched in three months, either the project is dead and should be archived or it is stalled and needs attention.

Progressive Summarization

Progressive summarization is Forte’s technique for distilling captured notes over multiple passes, each pass adding a higher layer of compression.

Pass 1 - Raw capture:
  Full article saved, quotes highlighted, minor annotations

Pass 2 - Bold pass:
  Most important sentences/phrases bolded within the highlights

Pass 3 - Summary:
  2-3 sentence executive summary written at the top of the note

Pass 4 - Remix:
  Ideas from this note appear in your own writing/thinking

The value of progressive summarization is that you invest additional attention in a note only when you return to it — when you are retrieving it because it is relevant to something you are working on. The first pass is cheap. The later passes happen organically as the note proves its worth. Notes that are never worth returning to never get the expensive later passes. The system self-selects for valuable material through actual use rather than predicted use.

The critique: most people complete Pass 1 and never return. If you are not actively working on projects that pull related notes back into view, the progressive summarization technique has no trigger to fire. This is why PARA and progressive summarization are designed together — the project folders are what create the retrieval pressure that drives the distillation passes.

Output Is the Point

This is worth stating plainly because PKM culture obscures it. The measure of a knowledge management system is not the size of the vault, the elegance of the graph, or the sophistication of the tagging system. It is the quality and volume of outputs the system enables. Blog posts, documentation, decisions made with better information, code written with less re-research, presentations with more substance. If your PKM does not generate more or better outputs than you would produce without it, it has delivered no value.

This framing is uncomfortable because outputs are hard to produce and systems are easy to build. But keeping the output objective in view is the only reliable protection against the library-building trap.

Obsidian as a Local-First PKM

Why Local Markdown Wins

The choice of PKM tool matters less than the practice, but it matters some. The properties worth optimizing for in a PKM tool are longevity, portability, and friction. Obsidian wins on all three for most technical users.

Obsidian stores notes as plain Markdown files in a local folder on your machine. This has several consequences that compound over time. Your notes are readable and editable by any text editor, on any operating system, in perpetuity — the format cannot become inaccessible because the company pivoted or shut down or decided to change their pricing model. You can process them programmatically with any text processing tool. You can put them in version control. You can move them between operating systems without conversion. You own them in a meaningful sense that cloud-native notes products do not enable.

The longevity argument is underrated. A PKM system is ideally something you maintain for years or decades. Evernote users who built their systems in 2010 experienced a deterioration in product quality and a pricing restructure that pushed many of them through a painful migration. Notion users have built elaborate relational databases that are deeply tied to Notion’s proprietary format — a migration path exists but it is not seamless. Plain Markdown files are immune to this. They will be readable in 2045 by software that does not yet exist.

Privacy is the other structural advantage. Your notes contain your thinking, your research, your opinions, your professional context, and often sensitive information about projects, people, and decisions. A local vault does not send any of that to a third party by default.

Core Obsidian Features That Matter

Wikilinks — [[note title]] syntax that creates links between notes — are the foundation of the graph structure. Creating a link is low-friction enough that you actually do it while writing, and the backlink panel on any note shows you everything that links to it. Backlinks are often more useful than forward links because they reveal connections you did not plan when you created the note.

The graph view renders your vault as a visual network of nodes and links. It is genuinely useful occasionally — a tightly connected cluster of notes reveals a topic you have thought about extensively; an isolated note signals something that has not been integrated into your thinking. But the graph view is primarily an orientation tool rather than a navigation tool. You will not spend most of your time in it, and if you find yourself spending a lot of time in it, you are probably optimizing aesthetics rather than doing knowledge work.

Templates reduce the cost of creating structured notes. A meeting notes template, a project kickoff template, a book note template — these lower the friction of starting any recurring note type and ensure you capture the fields that prove useful for later retrieval.

Daily notes give you a dated journal entry for each day with zero setup cost. They serve as a capture buffer when you do not have time to decide where something belongs, as a log of what you worked on, and as a location for ephemeral notes that do not need a permanent home. Many people find that daily notes, plus search, constitute most of the value they extract from Obsidian.

Canvas provides a spatial workspace for arranging notes, ideas, and connections visually. It is useful for planning projects, working through architectural decisions, and building visual explanations. Like the graph view, it becomes a distraction if you use it as the primary interface rather than an occasional tool.

The Plugin Ecosystem

Obsidian’s core functionality is deliberately minimal. The plugin ecosystem is where it becomes a platform. At the time of writing, the community plugin directory contains over a thousand plugins, which is both its strength and its surface area for scope creep. A few plugins are genuinely load-bearing for most serious users:

The discipline with plugins is the same as with note organization: add them when you have a specific friction they resolve, not because they look interesting. Every additional plugin is an additional cognitive surface to maintain.

The Mobile Sync Problem

Obsidian’s local-first model creates a real friction point on mobile: because the vault is a folder of files, you need a sync mechanism to keep your phone and desktop in sync. The options are:

Obsidian Sync is the official paid solution ($10/month). It is end-to-end encrypted, handles conflict resolution gracefully, and integrates with the Obsidian mobile app with no configuration. It is the right answer for most people who want mobile access and are willing to pay for it.

iCloud Drive works for macOS/iOS users and is free. Performance is adequate for most vault sizes. The practical issue is that iCloud can cause sync conflicts if you write to the vault on multiple devices simultaneously, and the Obsidian iOS app needs to be open for sync to run.

Syncthing is the self-hosted option. It runs on your LAN and syncs the vault directory directly between devices. It requires more setup and does not handle offline-then-sync as gracefully as Obsidian Sync, but it keeps your data entirely on your own hardware. For homelab users who are already running Syncthing for other things, this is the natural choice.

AI Plugins in Obsidian

The Privacy Spectrum

Before going through specific plugins, it is worth being explicit about what leaves your machine with each approach, because the privacy implications vary significantly.

Local-only (no network calls):
  Smart Connections (default config, local embeddings)
  |-- Your notes stay on your machine
  |-- Embeddings computed by WebAssembly transformer model
  |-- No API key required

Cloud API (data leaves your machine):
  Copilot for Obsidian (requires OpenAI / Anthropic / Gemini key)
  Smart Connections (optional: when you use chat with cloud LLM)
  Text Generator (requires API key)
  |-- Note content sent to third-party LLM provider
  |-- Governed by provider's data handling terms
  |-- Retrieval quality generally higher

Self-hosted LLM (local model, local API):
  Smart Connections + Ollama
  Copilot for Obsidian + Ollama endpoint
  |-- Model runs on your hardware
  |-- No cloud dependency
  |-- Requires beefy hardware for acceptable performance

If your vault contains sensitive professional information — client details, unreleased product plans, security findings, personal health data — the local embedding path via Smart Connections is the right default. The chat quality is lower than a frontier model, but the privacy posture is meaningfully different.

Smart Connections: Local Semantic Search

Smart Connections (developed by Brian Petro, ~786,000 Obsidian plugin downloads as of early 2026) is the most important AI plugin in the ecosystem for one specific reason: it is the only option that provides semantic search with no cloud dependency and no API key by default. It runs transformer embedding models via WebAssembly in Obsidian itself, computes embeddings for every note in your vault, stores them locally, and uses them to answer two questions that matter enormously in a PKM system: “what notes are related to this one?” and “what do I already know about this topic?”

The setup experience is intentionally minimal. You install the plugin, it indexes your vault in the background (a few minutes for vaults under a few thousand notes), and a “Similar Notes” panel appears in the sidebar showing semantically related notes as you navigate. You do not configure anything to get this working.

The chat interface allows you to ask questions in natural language and receive answers grounded in your vault content. The default local model is competent for retrieval but produces lower-quality synthesis than a frontier model. Smart Connections also supports optional integration with cloud LLMs (Claude, Gemini, ChatGPT, Llama 3 via API) for higher-quality chat while keeping the embedding and retrieval local.

The core workflow that makes Smart Connections genuinely valuable:

You're writing a note on distributed tracing.
Smart Connections sidebar shows:
  - "observability-fundamentals" (0.89 similarity)
  - "opentelemetry-setup-notes" (0.86 similarity)
  - "sre-incident-review-2025-11" (0.71 similarity)

You had forgotten you took notes on that incident.
The incident notes contain context about your specific
tracing gaps that is directly relevant to what you're
writing. You link them and incorporate the context.

This is the legitimate AI value-add in PKM: not generating text you did not write, but surfacing your own past thinking at the moment it is relevant. That is a genuinely hard problem that embeddings solve well.

Copilot for Obsidian: LLM Chat Over Your Vault

Copilot for Obsidian (developed by Logan Yang) takes a different approach: it is an LLM chat interface that lives inside Obsidian and can use your vault as context. You provide an API key for a supported provider — OpenAI, Anthropic Claude, Google Gemini, and others — and gain a chat panel that can answer questions by retrieving relevant vault notes and synthesizing responses.

The Vault Q&A feature indexes your entire vault for semantic search and provides answers with citations to the specific notes used. Unlike Smart Connections, which shows related notes passively, Copilot answers questions directly. You ask “what are my notes on blue-green deployments?” and receive a synthesized answer with links to the source notes rather than a list of related notes to read yourself.

As of version 3.1, Copilot added long-term memory as an agent tool and agentic capabilities for Plus subscribers. The practical use cases for agentic Copilot in a PKM context are narrower than the marketing implies — the most useful capability is “compose a draft using my existing notes on this topic” as a starting point for writing, not autonomous note organization.

The meaningful limitation: Copilot requires a cloud API key and sends note content to the API provider. The context window constraints of the underlying model also limit how much vault content can be included in any single query. For large vaults, smart retrieval (finding the right notes to send as context) becomes the bottleneck, and the quality of answers depends heavily on how well the retrieval identifies the relevant notes.

Text Generator: Inline AI Completion

Text Generator provides inline AI completions and template generation directly in the editor. Rather than a separate chat panel, it generates text in-place — you write a bullet-point outline, invoke Text Generator, and it expands the outline into prose. You write a question at the top of a note and invoke it to generate a structured answer as a starting point.

The practical value is as a drafting accelerator, not as a knowledge retrieval tool. Text Generator does not understand your vault context by default — it is calling an API with whatever text is near your cursor. The plugin supports custom templates, which makes it useful for recurring generation patterns: meeting summary templates, weekly review scaffolding, prompt templates for your most common writing tasks.

It is the right tool when you want inline generation without switching to a chat interface. It is less relevant if you are primarily using AI for knowledge retrieval rather than drafting assistance.

Building a Personal Knowledge Graph

Atomic Notes and Why Density Matters

The phrase “one idea per note” is the atomic note principle, and it matters more than it initially sounds. Notes that contain multiple distinct ideas resist linking — when you try to link to a note from elsewhere, you end up linking to a container that holds both the relevant idea and two unrelated ones. When you search for a concept, you find the right note but have to hunt within it. When you try to build a graph that reflects your thinking, the nodes are too coarse-grained to reveal interesting structure.

Atomic notes are also easier to progressively summarize, easier to link bidirectionally, and easier to resurface in context via semantic search. A note called “CRDT conflict resolution” is a more useful retrieval target than “Distributed Systems Study Notes — Week 3.”

The cost of atomic notes is overhead in note creation. Writing five atomic notes instead of one big note takes more time upfront. The payoff is in retrieval quality over time. For practical purposes, “atomic” does not mean maximally granular — it means one coherent idea or concept, expressible in a sentence. A four-paragraph note on how CRDTs handle merge conflicts is atomic. A twelve-paragraph note on distributed systems covering CAP theorem, CRDTs, consensus algorithms, and eventual consistency is not.

Maps of Content

A Map of Content (MOC) is a note whose primary content is links to other notes on a topic, serving as a navigational layer above the flat note structure. Rather than organizing notes into folders by topic (which is static and forces you to choose a primary category for every note), MOCs create a navigable index that can be created on demand, updated over time, and can link to notes that live in multiple contexts.

[[Distributed Systems MOC]]
  |
  +-- Fundamentals
  |     - [[CAP Theorem]]
  |     - [[Consistency Models]]
  |     - [[Network Partitions]]
  |
  +-- Consensus
  |     - [[Paxos Explained]]
  |     - [[Raft Algorithm]]
  |     - [[Practical BFT]]
  |
  +-- Data Structures
  |     - [[CRDT Types and Use Cases]]
  |     - [[Vector Clocks]]
  |
  +-- My Experience
        - [[incident-2025-07 split-brain postgres]]
        - [[migration-notes-kafka-clustering]]

The MOC can link to notes in your Projects folder, your Resources folder, and your daily notes simultaneously. The map is a perspective, not a container. This is the key structural advantage of MOC-based organization over folder-based organization: a note can appear in multiple MOCs without being duplicated, which is impossible in a strict folder hierarchy.

Tagging: The Proliferation Trap

Tags in Obsidian behave like labels that can be applied to any note and queried across the vault. The failure mode is well-documented and nearly universal: after a year of tagging, you have 200 tags, many used on fewer than three notes, with overlapping meanings and inconsistent application. The tags are no longer useful for navigation and cannot be meaningfully maintained.

The practical approach that works:

What tags are for:
  - Note TYPE: #daily-note, #meeting, #project-note, #reference, #inbox
  - Note STATUS: #draft, #published, #archived, #needs-review
  - Domain (broad, 5-10 max): #ops, #dev, #homelab, #learning

What tags are NOT for:
  - Every topic the note touches (use links instead)
  - Metadata that belongs in frontmatter properties
  - Organizing notes you don't know how to organize

The rule of thumb: if you could use a link instead of a tag, use the link. Tags are for orthogonal metadata (type, status, domain) that does not belong in the note graph structure. Topic relationships belong in links.

Dataview and Obsidian Bases: Querying Your Knowledge Base

Dataview is a community plugin that treats your vault as a database, allowing you to write SQL-like queries over note properties and content. Obsidian Bases (launched as a core plugin in Obsidian 1.9) provides a more approachable UI for the same underlying concept — table, card, list, and map views over notes filtered by properties.

The combination is powerful for the kinds of metadata queries that matter in project management and review:

TABLE file.mtime AS "Last Modified", status, tags
FROM "Projects"
WHERE status != "archived"
SORT file.mtime DESC

This surfaces your active projects sorted by when you last touched them — a useful weekly review starting point. Similarly:

LIST
FROM #needs-review
SORT file.ctime ASC
LIMIT 20

Surfaces your oldest unreviewed notes for a cleanup pass. The practical value of Dataview is not in the sophistication of the queries but in the specific patterns that become useful habits: weekly reviews that surface stale projects, periodic passes through the inbox, tracking reading lists and their status.

Obsidian Bases provides the same capability through a more visual interface — a .base file defines a view that can be embedded in any note, showing a filtered, sorted, formatted table of notes matching the criteria. For users who prefer not to write Dataview syntax, Bases is now the recommended approach.

RAG Over Your Own Notes

What Retrieval-Augmented Generation Means in the PKM Context

Retrieval-augmented generation (RAG) is a technique where you retrieve relevant documents from a corpus and include them as context in a prompt to a language model, allowing the model to answer questions grounded in your specific documents rather than in its general training data. In the PKM context, “your documents” are your notes, and the result is a system where you can ask natural language questions and receive answers synthesized from your actual writing and research.

The architecture is straightforward:

Your Notes (Markdown files)
        |
        v
  [Embedding Model]
  Converts text chunks to
  dense vector representations
        |
        v
  [Vector Database]
  Stores vectors + metadata
  (note title, path, chunk text)
        |
        v
  [Query Interface]
  "What do I know about
   blue-green deployments?"
        |
        v
  [Semantic Search]
  Query converted to vector,
  top-k similar chunks retrieved
        |
        v
  [LLM Synthesis]
  Retrieved chunks + query
  sent to language model
        |
        v
  Answer with citations
  to source notes

The key property that distinguishes this from a simple full-text search is that semantic search finds conceptually related content even when the query does not share vocabulary with the notes. “How do I handle a split-brain scenario?” will retrieve notes that discuss network partition handling, quorum consensus, and split-brain recovery procedures even if none of them use the phrase “split-brain scenario” in those exact words.

The Technical Stack Options

Smart Connections (easiest): Zero infrastructure. Runs inside Obsidian. Local WebAssembly transformer model for embeddings. Suitable for vaults under ~10,000 notes with minimal setup. The right choice for most individual users who want local RAG without running separate services.

Khoj (self-hosted, feature-complete): Khoj is an open-source self-hosted AI assistant backed by Y Combinator that indexes Markdown notes, PDFs, plain text files, and GitHub repos, provides an Obsidian plugin that syncs your vault to the Khoj instance, and supports both local LLMs via Ollama and cloud API providers. It provides a web interface, an Obsidian chat panel, and an API. For users who want RAG over not just their Obsidian vault but also their full document archive and external sources, Khoj is the most complete self-hosted option.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# docker-compose.yml excerpt for Khoj self-hosted
services:
  khoj:
    image: ghcr.io/khoj-ai/khoj:latest
    ports:
      - "42110:42110"
    volumes:
      - ~/.khoj:/root/.khoj
      - ~/notes:/notes:ro    # Mount your notes directory read-only
    environment:
      - KHOJ_ADMIN_EMAIL=you@example.com
      - KHOJ_ADMIN_PASSWORD=changeme

Custom pipeline with Ollama + ChromaDB/Qdrant: For users who want full control over the pipeline — chunk size, embedding model, query logic, output format — the custom approach gives you flexibility at the cost of significant setup time.

Obsidian vault (Markdown files)
    |
    v
Python script (LangChain or llama-index)
  - Walk vault directory
  - Chunk notes by paragraph / header
  - Embed each chunk with Ollama (nomic-embed-text or Qwen3-Embedding)
  - Store in ChromaDB (simpler) or Qdrant (scales better)
    |
    v
Query interface (CLI script or simple web UI)
  - User query -> embed with same model
  - Top-k nearest chunks from vector DB
  - Assembled as context for local LLM (Ollama: llama3, qwen2.5, etc.)
  - Response with source citations

Embedding Model Selection

For a personal knowledge base under 5,000 notes on consumer hardware, nomic-embed-text is the correct default. It runs acceptably on CPU, its 8K context window handles most full notes without chunking, and its retrieval quality is good enough that the differences on the benchmark leaderboard do not translate to perceptible differences in daily use. If you have an NVIDIA GPU available and retrieval precision matters — for a large professional document archive — Qwen3-Embedding is worth the additional resource cost.

One critical constraint: vectors from different models are incompatible. If you change your embedding model, you must re-embed your entire corpus. Pick one model and use it for everything, or you cannot run similarity search across the full collection.

The Quality Floor

RAG is only as good as the notes it operates over. This is the fact most RAG-in-PKM content conveniently skips. If your notes on a topic are sparse, the retrieval step will return thin context. If your notes are copy-paste extracts without your own synthesis, the model will synthesize someone else’s thinking, not yours. If your notes are poorly structured, the chunking step will split ideas at the wrong boundaries and the retrieval quality will suffer.

The quality of the RAG output is constrained by the quality of the note-taking beneath it. AI cannot compress good ideas out of mediocre notes. What it can do is help you find the good notes you have already written and connect them to what you are currently working on.

Note-to-Document Pipelines

Progressive Elaboration

The most reliable workflow for turning PKM notes into published documents is progressive elaboration: starting with the atomic notes you have accumulated on a topic, building an outline from them, expanding the outline into a rough draft, and then polishing the draft into a final document. Each stage is a discrete task with a clear completion state.

Stage 1: Atomic notes
  - [[load-balancer-algorithms]]
  - [[round-robin-limitations]]
  - [[least-connections-algorithm]]
  - [[consistent-hashing-intro]]
  - [[session-affinity-tradeoffs]]

Stage 2: Outline (from notes + gaps analysis)
  - Introduction: why load balancer algorithm matters
  - Round robin: simple but stateless
  - Least connections: better for variable request cost
  - Consistent hashing: for stateful services
  - Session affinity: when you cannot avoid state
  - Choosing: decision tree based on service characteristics

Stage 3: Rough draft
  - Expand each outline section using note content
  - Write transitions between sections
  - Note where content is thin (gaps)

Stage 4: Polish
  - Tighten prose
  - Add examples and diagrams
  - Verify technical accuracy
  - Publish

AI is useful in Stages 2 and 3 of this workflow. At Stage 2, asking “given these notes, what gaps exist in my coverage?” surfaces topics you have not addressed. At Stage 3, asking “expand this bullet point into two paragraphs in my existing tone” is a fast way to get a draft started without starting from a blank page.

AI Gap Analysis

A practical prompt for gap analysis using your notes as context:

I'm writing a guide on [topic]. Here are my current notes:

[paste relevant note content]

What important aspects of [topic] are missing from
these notes that a complete guide would need to cover?
List them as specific topics I should research.

This is a legitimate use of AI in PKM: using the model’s broad knowledge to identify blindspots in your own research, producing a to-do list for additional note-taking rather than generating the content directly.

Publishing From Obsidian

Obsidian Publish is the native publishing option — a paid service ($10/month) that hosts a public-facing website from selected vault notes, respecting wikilinks and backlinks in the rendered HTML. Suitable for digital gardens and personal wikis where the hyperlink structure is part of the design.

Hugo/Jekyll integration is the approach for users running their own blog (including this one). Obsidian’s Markdown is largely compatible with Hugo’s content format. The workflow is either a symlink from the Obsidian vault to the Hugo content directory, or a script that copies and transforms notes for publishing. The main friction is frontmatter — Obsidian uses frontmatter for its own metadata, and you need to ensure the published frontmatter is clean.

Pandoc conversion is the right tool when the output target is not a website — when you need to produce a PDF, a Word document, or a LaTeX source from your Markdown notes. Pandoc handles Markdown-to-anything conversion competently, and a shell script running pandoc over a note or set of notes provides a clean export pipeline.

When PKM Becomes Procrastination

The Symptoms

The moment at which knowledge management has become procrastination is identifiable and most people who have been in the PKM ecosystem for a year or more have experienced it. The symptoms are specific enough to diagnose honestly.

You are spending more time organizing than thinking. Your weekly review is two hours of moving notes between folders and applying tags, with no time remaining to write anything or review what you have captured.

You are building elaborate tagging systems for notes you will never re-read. You have a taxonomy for processing states, a taxonomy for media types, a taxonomy for confidence levels, a taxonomy for domain areas. The tags are internally consistent. The notes themselves are still not being used.

You are tool-hopping. You have read that Logseq’s outliner model is better for your thinking style. You spent a weekend migrating your vault. You are now reading about Roam’s block reference features and wondering if you should migrate again. The content of your knowledge base is unchanged; only the tooling has evolved.

You are reading about PKM instead of doing work. There is a specific infinity scroll of PKM YouTube channels, newsletters, subreddits, and communities that will fill an unlimited amount of time with content about improving your note-taking system. If you are consuming this content instead of using your system, the content has become part of the problem.

The common thread is that all of these activities feel productive — they involve effort, they produce something, they concern themselves with knowledge and organization — but they do not move the actual work forward. They are displacement activities with an intellectual patina.

The 80/20 Reality

In most PKM systems that have been maintained for a year or more, the majority of the value extracted from the system comes from a small fraction of the notes. Usually: active project notes (the highest value per note), a handful of durable reference pieces (command references, decision templates, personal runbooks), and the daily notes that serve as a searchable journal of recent work.

The elaborate resource library — the thousands of notes on topics you read about and captured but never returned to — delivers almost nothing in practice. The value-to-effort ratio of those notes is negative once you account for the organization overhead they imposed.

This suggests a heretical simplification: most of what belongs in a PKM system is not the comprehensive resource library that the second brain mythology promises, but rather a well-maintained project workspace and a searchable journal of recent work. Everything else is gravy.

The Flat Note-Taking Argument

There is a serious case for flat note-taking that the PKM community consistently undersells. Flat note-taking means: every note goes in a single folder, with a date-stamped or descriptively-named filename, and retrieval happens entirely via full-text search. No hierarchy, no tags, no links, no graph. Just notes and search.

The argument is not that organization is bad. The argument is that the value of organization needs to exceed the cost of maintaining it, and for most people it does not. Modern full-text search across a folder of Markdown files is fast and accurate. Tools like ripgrep, Obsidian’s built-in search, and Alfred can find anything in a vault of thousands of notes in under a second. If you remember roughly what you wrote and roughly when, you can find it without any organizational infrastructure.

The people who benefit most from structured PKM systems are those who write in it frequently enough that search becomes genuinely noisy, who have enough overlapping topics that link-following is faster than search, and who have the discipline to maintain the organizational overhead consistently over years. This is a narrower population than PKM content implies.

The Minimum System That Delivers 80% of the Value

If you want a PKM system that actually delivers value without consuming the time and energy of a second job, this is the minimum setup that works:

Daily notes as primary capture. Every day, a new note dated YYYY-MM-DD. Anything worth remembering goes there with no organization decision required. Use it as a log of what you are working on, a capture buffer for things to process, and an inbox for links, quotes, and half-formed ideas.

One note per active project. A single note for each project you are actively working on. No elaborate folder structure — just a note with the project name, current status, next actions, and links to relevant references. When the project is done, move it to an archive folder.

A handful of durable reference notes. Maintained actively, linked from wherever you use them. Not a comprehensive topic library — just the handful of references that you actually return to repeatedly.

Full-text search as the primary retrieval mechanism. Find things by what they contain, not where they live.

Smart Connections for surfacing what you forgot. Let the local embeddings show you what notes are related to what you’re currently writing. This is the AI addition that actually adds value without adding system complexity.

This is not a glamorous system. It does not produce a beautiful knowledge graph or a second brain dashboard with satisfying completion metrics. It produces notes you can find, outputs you actually write, and a practice sustainable over years rather than months.

The knowledge compounds not because you built the right system, but because you showed up to it consistently. Consistency requires low friction. Low friction requires simplicity. Most PKM systems fail because they prioritize depth of organization over sustainability of practice.

Build the simplest system you will actually maintain. Then maintain it for long enough that the compounding begins.

Practical Setup Summary

For a new Obsidian-based PKM with AI integration, this is the order in which setup delivers value:

1. Create your vault in a folder you back up and (optionally) sync. Plain folder. No structure yet.
2. Enable daily notes in core plugins. Configure them to go in a Daily Notes/ folder, formatted YYYY-MM-DD.
3. Install Smart Connections. Let it index. Check the similar notes panel. That’s it for AI — you can get value immediately.
4. Start capturing. Write in your daily note. Create notes for active projects. Do not organize yet.
5. After 30 days, look at what you have and where the friction is. Add Dataview or Bases if you need to query properties. Add a simple folder structure (inbox, projects, resources, archive) if search is getting noisy. Add Copilot for Obsidian if you want LLM chat with frontier model quality.
6. Only after your vault contains real work consider adding: Khoj for multi-source RAG, a custom Ollama pipeline for more control, or publishing integrations.

The AI layer is worth adding early — Smart Connections specifically, because it is zero-friction and requires no API key. The structural layer should follow the actual shape of your usage, not precede it.

PKM content has a vested interest in the complexity of PKM systems. The more elaborate the framework, the more content there is to produce about it, and the more compelling the tools, courses, and consulting that surround it. The audience for that content — intellectually curious people who feel behind on synthesizing what they know — is perpetually susceptible to the promise that the right system will finally unlock the value of their accumulated reading.

The systems are real and some of them are good. Obsidian is genuinely excellent software. Smart Connections is a legitimate advancement in how personal knowledge becomes retrievable. RAG over your own notes with local embeddings is a real capability that did not exist a few years ago and is worth having.

But none of it substitutes for writing. The second brain is not a storage system. It is a drafting environment. The measure is what you produce with it, not what it contains.