LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

The Lisp Story

21 min read Computing History
lispschemecommon-lispclojurefunctional-programmingprogramming-languagesmacros
Contents

Lisp is the second-oldest programming language still in use, and it has spent most of its sixty-eight years being simultaneously ahead of its time and commercially marginal. John McCarthy did not set out to design a practical language; he was trying to write down a clean mathematical notation for symbolic computation, and the language fell out of the notation almost by accident. That origin explains nearly everything strange and durable about Lisp: its parenthesized syntax is a direct consequence of representing programs as data, its evaluation model was specified as a short recursive function before any compiler existed, and its commercial history is a long sequence of brilliant systems that lost to cheaper, dumber hardware. The remarkable thing is not that Lisp failed to take over the world. It is that almost every idea Lisp pioneered — garbage collection, the read-eval-print loop, first-class functions, closures, dynamic typing with late binding, conditional expressions, and metaprogramming through macros — quietly took over the world anyway, embedded in languages that look nothing like Lisp. This is the story of how that happened, who paid for it, and why a sixty-year-old idea keeps coming back.

McCarthy and the Notation That Became a Language

In 1958 John McCarthy was at MIT, working on artificial intelligence problems that required manipulating symbolic expressions rather than numbers — differentiating algebraic formulas, proving theorems, reasoning about lists of facts. FORTRAN existed and was good at arithmetic, but it had no clean way to represent and transform tree-structured symbolic data. McCarthy wanted a notation grounded in recursive function theory, drawing on Alonzo Church’s lambda calculus and the theory of recursive functions.

His 1960 paper, Recursive Functions of Symbolic Expressions and Their Computation by Machine, laid out the whole thing. The data model was the S-expression (symbolic expression): an atom, or a pair of S-expressions written (a . b), with lists built from nested pairs. The genius move was that programs would be written in exactly the same notation as data. A function call (f x y) is just a list whose first element names the operation. This property — code and data sharing one representation — is homoiconicity, and it is the single fact from which most of Lisp’s later character derives.

McCarthy then did something nobody had done before: he wrote the interpreter for the language in the language itself, as a function called eval. This metacircular evaluator was meant as a theoretical demonstration — a proof that the language was self-describing — not as something to run. Steve Russell, one of McCarthy’s graduate students, noticed that the eval function in the paper was an actual specification of an interpreter and hand-compiled it into IBM 704 machine code. McCarthy later recalled being skeptical until Russell showed him it ran. Lisp existed as a working language because someone implemented the mathematics literally.

The core of eval is small enough to fit on a page. Here it is in something close to McCarthy’s original form, written in modern Common Lisp, evaluating a tiny Lisp inside Lisp:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

(defun my-eval (expr env)
  (cond
    ((symbolp expr) (cdr (assoc expr env)))          ; variable lookup
    ((atom expr) expr)                                ; self-evaluating literal
    ((eq (car expr) 'quote) (cadr expr))             ; (quote x) -> x
    ((eq (car expr) 'if)                              ; (if c t e)
     (if (my-eval (cadr expr) env)
         (my-eval (caddr expr) env)
         (my-eval (cadddr expr) env)))
    ((eq (car expr) 'lambda) expr)                    ; a lambda is its own value
    (t (my-apply (my-eval (car expr) env)             ; application
                 (mapcar (lambda (a) (my-eval a env)) (cdr expr))
                 env))))

(defun my-apply (fn args env)
  (let ((params (cadr fn))
        (body (caddr fn)))
    (my-eval body (append (pairlis params args) env))))

That this fits in twenty lines, and that those twenty lines are themselves valid data you can construct and transform at runtime, is the whole magic trick. Everything Lisp does later is an elaboration of eval and apply calling each other. The seven primitives McCarthy identified — quote, atom, eq, car, cdr, cons, cond — are sufficient to bootstrap the rest. Paul Graham would later call this the “Maxwell’s equations of software.”

McCarthy also introduced two ideas the rest of computing took decades to adopt. The first was the conditional expression — cond returning a value, rather than the goto-based branching of contemporary languages. The second was automatic memory management: because programs constantly built and discarded list structure, McCarthy’s team implemented the first garbage collector to reclaim unused cons cells. Every language that frees you from manual malloc/free traces back to this. Languages like C deliberately did not take that path, trading safety for control, and the two philosophies have been in tension ever since.

Maclisp, the AI Lab, and a Whole Operating System in Lisp

Through the 1960s and 1970s Lisp grew up inside the MIT Artificial Intelligence Laboratory, where the dominant dialect was Maclisp (named after Project MAC, not Apple). This was the same hothouse culture that produced the hacker ethos, the ITS operating system, and a relationship with the emerging Unix world that was equal parts kinship and rivalry. The Unix tradition prized small composable tools written in C; the Lisp tradition prized a single live, malleable image you could reshape from the inside. Both were reactions against the batch-processing mainframe world, and they clashed culturally for decades.

Maclisp was fast for its time and accumulated a sprawl of features, but the AI Lab’s ambitions kept outrunning the hardware. Lisp programs wanted enormous address spaces, hardware-checked tags to distinguish pointers from integers, and runtime support for garbage collection — none of which general-purpose minicomputers provided. So the AI Lab did the audacious thing: it designed computers whose machine architecture was tailored to run Lisp.

The Lisp Machine was a single-user, networked, bitmapped-display workstation with a tagged architecture, hardware support for garbage collection, and an operating system written almost entirely in Lisp — all the way down. You could trace a bug from your application code into the windowing system into the network stack into the disk driver, and it was all inspectable, editable, redefinable s-expressions in a running image. There was no meaningful line between “the system” and “your program.” For the people who used them, Lisp Machines were the most productive development environments that had ever existed, and arguably remained so for an embarrassingly long time. The Emacs editor, the modern interactive debugger, and the integrated REPL workflow all matured in this environment.

The Boom: Symbolics, LMI, and the AI Gold Rush

In 1979 the AI Lab spun the technology out. The commercialization split, somewhat acrimoniously, into two companies: Lisp Machines, Inc. (LMI) and Symbolics. This split — and the question of whether the freely shared AI Lab software would be locked up by a company — was the proximate trigger for Richard Stallman’s launch of the GNU Project and the free software movement. The Lisp Machine wars are, in a real sense, the origin story of the GPL.

The early-to-mid 1980s were boom years. Artificial intelligence was the hot technology, expert systems were going to revolutionize business, and you needed a Lisp Machine to build them. Symbolics shipped genuinely advanced hardware and software: the 3600 series, the Genera operating system, sophisticated graphics, and development tools that would not be matched by mainstream platforms for fifteen years. Symbolics.com was, in 1985, the very first registered .com domain on the internet. A Lisp Machine cost tens of thousands of dollars, and for a few years customers paid it.

Era Dominant dialect / system Where it ran What it optimized for
1958–1962 McCarthy’s Lisp / Lisp 1.5 IBM 704 / 7090 Specifying symbolic computation at all
1966–1980 Maclisp PDP-6/10, ITS Raw speed and AI Lab research
1974–1980 Interlisp PDP-10, Xerox Programmer environment, “DWIM”
1980–1988 Zetalisp / Genera Symbolics, LMI hardware A complete Lisp operating system
1975–1990s Scheme Anywhere; teaching Minimalism, lexical scope, semantics
1984–today Common Lisp Commodity hardware Standardization, industrial use
2007–today Clojure JVM, then CLR/JS Immutability, concurrency, host reach

The Bust: When the Hardware Caught Up and the Money Left

Two things killed the Lisp Machine. The first was Moore’s Law working against a niche. Lisp Machines justified their price by doing in hardware what commodity machines could not do in software — tagged memory, fast garbage collection, large address spaces. But stock workstations from Sun got faster every eighteen months, and software implementations of Common Lisp on cheap hardware closed the gap and then erased it. A specialized $50,000 machine cannot compete with a general-purpose $5,000 machine that is almost as good and runs everything else too. This is the same dynamic that eventually killed every specialized-hardware bet against commodity silicon.

The second was the AI winter. The expert-systems boom had been oversold; the systems were brittle, expensive to maintain, and rarely delivered the promised returns. As corporate AI budgets collapsed in the late 1980s, demand for the expensive machines you built expert systems on collapsed with it. Lisp got tarred by association: it became “the AI language,” and when AI fell out of fashion, so did Lisp. Symbolics filed for bankruptcy in 1993; LMI was gone years earlier. The most advanced computing environment of its generation died because its business model was tied to a hardware premium that physics dissolved and a market that evaporated.

It is worth being honest about the technical lessons, not just the market ones. The Lisp Machine community optimized for a single-user, single-image, infinitely-malleable world at exactly the moment the industry was moving toward multi-user servers, file-based source control, and reproducible builds from text. A system where the running image is the program is wonderful for one expert hacker and a nightmare for a team trying to reconstruct a known-good state. Some of Lisp’s “marginalization” was the rest of the industry choosing properties — textual source of truth, separation of build and run, boring reproducibility — that the image-based world undervalued.

The Scheme Branch: Stripping Lisp to Its Essence

While the AI Lab was building ever-larger systems, a different impulse produced a radically smaller one. In 1975 Gerald Sussman and Guy Steele, at the MIT AI Lab, set out to understand the actor model of computation by writing a tiny Lisp interpreter. The result was Scheme, and in the process of building it they stumbled onto a discovery that reshaped how the whole industry thinks about variables.

Earlier Lisps used dynamic scoping, where a free variable’s value is looked up in the dynamic call chain at runtime. Scheme committed to lexical scoping, where a variable refers to the binding visible in the source text where the function was defined, not where it was called. This is what makes closures work properly: a function carries the environment of its definition with it. Lexical scope is now so universal — JavaScript, Python, Rust, every modern language — that it is hard to appreciate it was a research result. The “Lambda Papers” Sussman and Steele wrote argued, provocatively, that the lambda was a perfectly good universal control and data structure construct, and that fancy control flow could be expressed by passing functions around.

Scheme made several other commitments that became canonical:

  • Proper tail calls. A function call in tail position must not grow the stack. This makes iteration expressible as recursion without blowing up, and it is now a required feature of many languages (and a notably absent one in others, including the standard JVM).
  • First-class continuations via call/cc, which reify “the rest of the computation” as a callable value. This is the most powerful and most abused feature in the language; you can build coroutines, generators, exceptions, and backtracking from it, and you can also write programs no human can follow.
  • A tiny, precisely specified core. The Scheme reports (R5RS, R6RS, R7RS) are famous for fitting the entire language semantics in a small document.

Scheme’s largest cultural impact came through education. SICP — Structure and Interpretation of Computer Programs, by Abelson and Sussman — used Scheme to teach a generation of programmers to think about programs as data, evaluation as a process, and abstraction as the central activity of engineering. Here is the kind of thing SICP makes feel natural: a closure-based counter, demonstrating lexical scope and first-class functions in a few lines.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

(define (make-counter)
  (let ((count 0))
    (lambda ()
      (set! count (+ count 1))
      count)))

(define c (make-counter))
(c)  ; => 1
(c)  ; => 2

;; tail-recursive factorial: constant stack, thanks to proper tail calls
(define (factorial n)
  (define (iter acc k)
    (if (= k 0)
        acc
        (iter (* acc k) (- k 1))))   ; this call is in tail position
  (iter 1 n))

(factorial 20)  ; => 2432902008176640000

Common Lisp: Standardizing the Sprawl

By the early 1980s there were too many incompatible Lisps — Maclisp, Zetalisp, Interlisp, Spice Lisp, NIL, and more — and DARPA, which funded much of the work, wanted consolidation. A community effort, anchored by Guy Steele’s massive reference book Common Lisp the Language (1984) and culminating in the ANSI Common Lisp standard (X3.226) in 1994, unified the industrial dialects into one large, pragmatic language.

Common Lisp is unapologetically big. Where Scheme is a scalpel, Common Lisp is a fully stocked workshop: multiple number types including rationals and bignums, a condition-and-restart error system far more flexible than exceptions, the format directive mini-language, structured iteration via loop, packages for namespacing, and the crown jewel, CLOS.

The Common Lisp Object System is one of the most sophisticated object systems ever standardized, and it arrived as a library defined in the language rather than a built-in. Its central ideas — multiple dispatch (methods chosen on the types of all arguments, not just a privileged receiver), multiple inheritance with a well-defined linearization, and the Metaobject Protocol that lets you reprogram the object system itself — were decades ahead of mainstream OOP. Julia’s multiple dispatch and many later research languages are walking ground CLOS mapped in 1988.

But the feature that distinguishes Lisp from every language that merely borrowed its other ideas is the macro. Because code is data, a macro is a function that runs at compile time, receives unevaluated source as s-expression data, and returns new source to be compiled in its place. This is not text substitution like the C preprocessor; it is structural transformation over the actual parse tree, with the full language available to do the transforming. Here is a unless macro and a small swap written by hand, plus a REPL session showing it in action:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

;; A macro receives code as data and returns code as data.
(defmacro my-unless (condition &body body)
  `(if (not ,condition)
       (progn ,@body)))

;; ` is quasiquote, , unquotes a value, ,@ splices a list.
(macroexpand-1 '(my-unless hungry (eat) (sleep)))
;; => (IF (NOT HUNGRY) (PROGN (EAT) (SLEEP)))

;; A REPL session (SBCL):
CL-USER> (defparameter *x* 10)
*X*
CL-USER> (my-unless (> *x* 100) (format t "x is small~%"))
x is small
NIL
CL-USER> (loop for i from 1 to 5 collect (* i i))
(1 4 9 16 25)

Macros are why Lispers claim every other language is reinventing pieces of Lisp. Async/await, pattern matching, list comprehensions, embedded DSLs — in most languages these are features the designers must add and you must wait for. In Lisp they are libraries any user can write, because the user has the same code-as-data power the compiler does. This is the deepest trade-off in the language: that power is real, and it is also why two Lisp codebases can feel like two different languages, and why “just read the source” sometimes means “first learn the dozen macros this team invented.”

The Long Quiet Years

From the mid-1990s to the mid-2000s, Lisp was a respected backwater. Common Lisp implementations matured — CMUCL, then SBCL (Steel Bank Common Lisp), produced genuinely fast native code; commercial vendors like Franz (Allegro) and LispWorks served niche industrial customers in scheduling, theorem proving, CAD, and airline systems. Scheme thrived in academia and gave rise to Racket, an entire ecosystem for building languages.

But the mainstream had moved to Java and then to the dynamic-language wave of Python, Ruby, and JavaScript. Paul Graham’s essays in the early 2000s (“Beating the Averages,” the Viaweb story of using Common Lisp as a competitive weapon) kept a romantic flame alive and pulled in a steady trickle of curious programmers. The honest assessment of this period: Lisp had the best ideas and the worst distribution. It had no dominant implementation, no corporate champion, no batteries-included standard library by modern expectations, and no package manager that everyone used. Great languages do not win on merit; they win on ecosystems, and Lisp’s was fragmented across a dozen implementations and two major dialects.

The Clojure Revival: Lisp Meets the Boring Platform

In 2007 Rich Hickey, a working consultant who had spent years writing C++, Java, and C#, released Clojure after two years of self-funded work. Clojure is the most commercially successful Lisp since the Lisp Machine, and it succeeded precisely by refusing to be precious about the things older Lisps were precious about.

Hickey made a set of deliberate, opinionated bets:

  • Run on the JVM. Instead of fighting for its own platform — the strategy that died with Symbolics — Clojure embraced a host. It compiles to JVM bytecode, calls any Java library directly, and inherits the JVM’s garbage collector, JIT, and enormous library ecosystem. Later it gained ClojureScript (compiling to JavaScript) and a CLR port. This solved the distribution problem that had crippled every previous Lisp. The same “embrace the JVM” strategy underlies Scala, and the two languages represent opposite answers to the question of what a JVM functional language should look like.
  • Immutability by default. Clojure’s core data structures — lists, vectors, maps, sets — are immutable and persistent: an “update” returns a new structure that structurally shares almost all of its memory with the old one, so it is cheap in time and space. This is the engineering idea, drawn from Phil Bagwell’s hash array mapped tries, that makes pervasive immutability practical rather than ruinous.
  • A principled story for state. Where it needs mutable, coordinated state, Clojure provides software transactional memory (STM) via refs, plus atoms and agents for other concurrency patterns. Identity and value are separated: a value never changes, but a reference can be made to point at a new value, transactionally.
  • Data over ceremony. Clojure leans on a small set of generic data structures and functions over them, rather than a proliferation of classes. “It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures,” an old Lisp aphorism Hickey took seriously.

Clojure looks like Lisp and feels like something new. Here is the persistent-data and concurrency story in a few lines:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

;; Immutable, persistent data: assoc returns a NEW map, old one untouched.
(def base {:name "ada" :score 0})
(def updated (assoc base :score 10))
base     ; => {:name "ada", :score 0}      (unchanged)
updated  ; => {:name "ada", :score 10}

;; Coordinated mutable state with an atom (compare-and-swap, lock-free):
(def counter (atom 0))
(swap! counter inc)   ; => 1
(swap! counter + 41)  ; => 42

;; Same homoiconic macro power, modern syntax. A threading macro
;; expands into nested calls at compile time:
(->> (range 1 6)
     (map #(* % %))
     (filter even?)
     (reduce +))      ; => 4 + 16 = 20

It is instructive to put the three living branches of Lisp side by side on the same task — squaring a list and summing the even results — to see how much shared DNA survives across fifty years and three very different design philosophies:

1
2
3
4
5
6
7
8

Common Lisp:  (reduce #'+ (remove-if #'oddp
                  (mapcar (lambda (x) (* x x)) '(1 2 3 4 5))))

Scheme:       (fold-left + 0 (filter even?
                  (map (lambda (x) (* x x)) '(1 2 3 4 5))))

Clojure:      (reduce + (filter even?
                  (map #(* % %) [1 2 3 4 5])))

Hickey backed the language with a series of talks — Simple Made Easy, The Value of Values — that became influential well beyond Clojure, reframing how a broad swath of programmers think about simplicity, state, and identity. Clojure found a real commercial home in data engineering, backend services, and finance, and remains a working, paid-for language in 2026.

The honest trade-offs persist. JVM startup time is real, though GraalVM native images and Babashka mitigate it. Stack traces through macro-heavy, lazy, dynamically typed code can be miserable. The flexibility that lets a team build the perfect DSL also lets them build an inscrutable one. And immutability, while a superpower for reasoning and concurrency, has a cost you sometimes claw back with transients and primitive arrays in hot loops.

The Diffuse Influence: Lisp Won by Dissolving

Here is the paradox to sit with. Lisp the platform lost. Lisp the dialect family is, by market share, a rounding error. And yet Lisp’s ideas are now the water everyone swims in.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

            Lisp idea (year)                Where it lives now
   ----------------------------------    --------------------------------------
   Garbage collection (1959)        -->  Java, Go, Python, JS, C#, Swift...
   Conditional expressions (1960)   -->  the ?: and if-expression in everything
   First-class functions (1960)     -->  every language with a "lambda"
   The REPL / interactive eval      -->  python -i, node, irb, ghci, jshell
   Lexical closures (Scheme '75)    -->  JS, Python, Rust, Kotlin, Swift...
   Proper tail calls (Scheme '75)   -->  Scheme, Lua, some JS engines, Elixir
   Dynamic typing + late binding    -->  Python, Ruby, JavaScript
   Macros / code-as-data            -->  Rust macros, Elixir, Julia, Template Haskell
   Persistent data structures       -->  Clojure, Scala, Immutable.js, Erlang
   Multiple dispatch (CLOS '88)     -->  Julia, Dylan, multimethods everywhere

Every time you write an arrow function in JavaScript, you are using a lexical closure that Scheme had to invent. Every time you fire up python -i or a Node REPL, you are using McCarthy’s read-eval-print loop. Every garbage-collected language is paying a debt to a 1959 list reclaimer. The functional-programming renaissance of the last fifteen years — first-class functions, immutability, map/filter/reduce as the default vocabulary of data transformation, the idea that you reason about a program by reasoning about values — is, viewed from a sufficient height, the mainstream slowly rediscovering what Lisp asserted in 1960. Languages like Haskell, Scala, Roc, Elixir, and the functional corners of Rust and Swift are all downstream of ideas Lisp shipped first.

The one thing that has not diffused out is the macro — the full code-as-data power. You can have closures without homoiconicity, garbage collection without s-expressions, a REPL without eval. But a true macro system requires that programs be data in the language’s own data structures, and almost no popular language is willing to pay the syntactic price of parentheses-everywhere to get it. That is the irreducible core that keeps people coming back: not nostalgia, but the fact that in a Lisp, the language is not a fixed artifact handed down by a committee. It is clay.

Verdict

Lisp is the clearest case in computing history of a technology that lost every battle and won the war. Its hardware companies went bankrupt, its market share is negligible, its syntax remains an acquired taste, and its commercial heyday ended with the AI winter of the late 1980s. Judged as a product, it is a cautionary tale about betting on specialized hardware and tying your fortunes to a hype cycle.

Judged as a set of ideas, it is the most influential language ever designed. McCarthy’s twenty-line eval is still the cleanest explanation of what evaluation is. Scheme’s insistence on lexical scope and proper tail calls became universal law. Common Lisp’s CLOS and condition system are still ahead of most production languages. And Clojure proved that the whole approach — immutability, code-as-data, interactive development — is not a museum piece but a competitive advantage on modern hardware, once you stop trying to own the platform and embrace someone else’s.

Should you learn it? Yes, and not for your résumé. Work through SICP in Scheme to understand evaluation, then write a few macros in Common Lisp or Clojure to feel what code-as-data actually buys you. You will come away a better programmer in whatever language pays your bills, because you will recognize how many of its features are Lisp ideas wearing different syntax — and you will have a clear-eyed sense of which of Lisp’s promises were real and which were the seductive over-reach that bankrupted Symbolics. Lisp did not need to take over the world. It only needed to be the place the good ideas came from, and at that it has no rival.

Sources

Comments