Shannon and Information Theory: The 1948 Paper That Named the Bit

June 10, 2026 15 min read Mathematics for Engineers

information-theorymathematicscompressionnetworkingmachine-learning

Contents

In July 1948, a 32-year-old mathematician at Bell Telephone Laboratories published a two-part paper in the Bell System Technical Journal titled “A Mathematical Theory of Communication.” It was not a hardware design. It was not an algorithm. It was a precise answer to a question that nobody had managed to ask cleanly before: what is information, and how much of it can you reliably move through a physical channel? The paper introduced the word bit, gave a formula for entropy that would reappear in thermodynamics, machine learning, gambling theory, and compression algorithms, and drew hard limits — limits that no engineering cleverness can breach — around what every communications system you have ever used can do. We are still building inside those limits.

Shannon’s paper is remarkable for being both practically useful and genuinely fundamental. It does not describe a specific modulation scheme or a particular code. It describes the shape of the problem, proves what is achievable in principle, and lets engineers work out the how. Bell Labs produced an unusual share of such papers; Shannon’s may be its most consequential.

Entropy as Surprise

Shannon needed a measure of information content that behaved sensibly. His starting point was intuitive: a message you already knew carries no information. A message that surprises you carries a lot. More precisely, the information content of an event with probability p should be high when p is low and zero when p is 1.

The unique function (up to a constant) satisfying this requirement plus the reasonable demand that information be additive for independent events is:

I(x) = -log2(p(x))   [bits]

A fair coin flip has probability 0.5, so a single flip carries -log2(0.5) = 1 bit of information. An event with probability 0.01 carries -log2(0.01) ≈ 6.64 bits. Certain events carry zero bits.

Entropy H is the expected value of this self-information — the average surprise per symbol from a source:

H(X) = -sum over all x of p(x) * log2(p(x))   [bits per symbol]

For a fair coin, this is:

H = -(0.5 * log2(0.5) + 0.5 * log2(0.5))
  = -(0.5 * (-1) + 0.5 * (-1))
  = 1 bit

Now load the coin so it comes up heads 90% of the time:

H = -(0.9 * log2(0.9) + 0.1 * log2(0.1))
  = -(0.9 * (-0.152) + 0.1 * (-3.322))
  = -((-0.137) + (-0.332))
  = 0.469 bits

The loaded coin carries less than half the information of a fair one. This matches intuition: if you already know the outcome is almost certainly heads, each flip tells you less. Maximum entropy for a two-outcome source is achieved at equal probabilities; any skew reduces it. For a source with n equally probable symbols, H = log2(n). A fair six-sided die has H = log2(6) ≈ 2.585 bits per roll.

Entropy of Common Sources

Source	Approximate Entropy
Fair coin flip	1.000 bit/symbol
Fair six-sided die	2.585 bits/symbol
English text (letter-level)	~4.1 bits/character
English text (word-level, accounting for context)	~1.0–1.5 bits/character
A single byte from /dev/urandom	8.000 bits/byte
AES-256 ciphertext block	~8.000 bits/byte
Typical JPEG pixel (after transform)	~1–2 bits/pixel
English speech (audible spectrum)	~50 bits/phoneme

The English text row deserves attention. At the letter level, ignoring context, English has about 4.1 bits per character because the 26 letters are not equally frequent — ’e’ appears far more often than ‘z’. But human language is heavily contextual; the letter following ‘q’ is almost always ‘u’. Shannon estimated the true entropy of English to be somewhere between 0.6 and 1.3 bits per character, a remarkable compression potential that modern language models have rediscovered.

The Source Coding Theorem

Shannon’s first major theorem formalizes the intuition above into a hard bound:

You cannot compress data to fewer bits per symbol than the entropy of its source, on average. You can always compress down to entropy with a suitable code.

This is the source coding theorem (also called the noiseless coding theorem). It says entropy is not just a measure of surprise; it is the fundamental unit of compressibility. A source producing symbols with entropy H bits/symbol requires at least H bits/symbol to represent losslessly, no matter how clever the code.

This is why gzip, zstd, bzip2, and every lossless compressor in existence is bounded by the same ceiling. A well-engineered compressor approaches entropy; it cannot go below it. In practice, real compressors fall short because they use finite-window models of the source — they cannot observe the full statistical structure of a file the way the theorem assumes — but the gap narrows as model quality improves.

The constructive half of the theorem is just as important: optimal codes exist. Huffman coding achieves entropy to within one bit per symbol; arithmetic coding can approach it arbitrarily closely. The method matters less than Shannon’s guarantee that the target is achievable.

Why You Cannot Compress Random Data

The source coding theorem has a corollary that surprises people the first time they encounter it: a perfectly random source cannot be compressed at all.

Uniformly random data has maximum entropy — every byte value is equally likely, so H = 8 bits/byte. By the source coding theorem, you need at least 8 bits to represent each byte. Any “compressed” version must be at least as large as the original. This is not a limitation of existing algorithms; it is a mathematical certainty. In fact, for any lossless compressor, there must exist files that grow when compressed (otherwise you could compress repeatedly until the file vanished).

This has a direct operational implication: compress before you encrypt, not after. Modern encryption produces output indistinguishable from random by design — that is a security requirement. Ciphertext has entropy approaching 8 bits/byte. Feeding it to gzip will make it slightly larger, not smaller. But plaintext — natural language, source code, JSON, log files — has much lower entropy and compresses well. The correct pipeline is:

plaintext -> compress -> encrypt -> transmit/store

Reversing the order wastes bandwidth and storage for zero security benefit. HTTPS, SSH, and TLS all compress before encrypting (or did, until CRIME and BREACH attacks on TLS-level compression forced a rethink — a reminder that applied cryptography can interact with information theory in non-obvious ways).

The same principle explains why you cannot compute whether a file is “already compressed” in general. The best practical test is to estimate the file’s entropy: if it is close to 8 bits/byte, further lossless compression will not help. Tools like ent and zstd --check-compressed do this.

Channel Capacity

Shannon’s second major contribution addresses a different question: not how compact you can represent a source, but how much information you can push through a physical channel that adds noise. The result is the Shannon-Hartley theorem:

C = B * log2(1 + S/N)   [bits per second]

Where:

C is channel capacity in bits per second — the maximum rate of reliable communication
B is the bandwidth of the channel in hertz
S/N is the signal-to-noise ratio (as a linear power ratio, not in decibels)

This is arguably the most important formula in telecommunications. It tells you the theoretical maximum throughput of any channel given its bandwidth and noise conditions. No modulation scheme, no error-correcting code, no DSP technique can exceed it. Engineers spend careers getting closer to it.

The channel model Shannon had in mind:

                     noise
                       |
                       v
  Source -> [Encoder] -> [Channel] -> [Decoder] -> Destination
               |                          |
               +--------- feedback -------+
                        (optional)

The encoder maps source symbols to channel symbols. The channel corrupts them with noise. The decoder attempts to recover the original message. Shannon’s theorem specifies the maximum rate at which the decoder can succeed.

Shannon-Hartley in Practice: Wi-Fi

To see why this matters for everyday engineering, consider a Wi-Fi channel. 802.11ac (Wi-Fi 5) uses 80 MHz channels in the 5 GHz band. In good conditions, a nearby client might see an SNR of around 30 dB. Converting:

S/N (linear) = 10^(SNR_dB / 10) = 10^(30/10) = 1000

C = 80e6 * log2(1 + 1000)
  ≈ 80e6 * log2(1001)
  ≈ 80e6 * 9.97
  ≈ 798 Mbps

That is the theoretical ceiling. Real 802.11ac with a single spatial stream uses up to 256-QAM with a 5/6 code rate, achieving around 433 Mbps at 80 MHz. MIMO with multiple spatial streams multiplies this by the number of streams (each spatial stream is effectively a separate channel); an 802.11ac 4x4 MIMO link can claim up to 1.73 Gbps in theory.

The gap between theoretical capacity and advertised speeds is not inefficiency — it is the cost of practical constraints: guard intervals, pilot subcarriers, overhead frames, the fact that SNR varies across the channel bandwidth, and the need for backward compatibility with older devices.

Wi-Fi 6 (802.11ax) and Wi-Fi 7 (802.11be) push closer to Shannon by using 1024-QAM and 4096-QAM respectively, at the cost of requiring better SNR to decode reliably. The headline speed improvements are largely applications of the Shannon-Hartley formula: more bandwidth (wider channels), more spatial streams, and higher-order modulation where conditions allow.

Channel Capacity Examples

Channel	Bandwidth	Typical SNR	Shannon Limit
Phone modem (POTS line)	3.4 kHz	~35 dB	~38 kbps
DSL ADSL2+	2.2 MHz	~20 dB	~22 Mbps
Wi-Fi 5 (80 MHz, 30 dB SNR)	80 MHz	30 dB	~798 Mbps
Wi-Fi 6E (160 MHz, 35 dB SNR)	160 MHz	35 dB	~2.7 Gbps
5G mmWave (400 MHz, 25 dB SNR)	400 MHz	25 dB	~1.3 Gbps
Fiber (single wavelength)	~4 THz	~30 dB	~40 Tbps

The phone modem row explains why 56k modems were the limit of the POTS era. V.92 at 56 kbps was within spitting distance of the Shannon limit for a standard telephone channel. Faster dial-up was not a solvable engineering problem; it was a physics problem. The industry moved to DSL, which widened the channel rather than asking more of the same bandwidth. Cell networks face the same tradeoffs at scale — each generation either widens channel bandwidth, increases spatial reuse, or adds modulation orders to harvest more bits per hertz from a given SNR.

The Channel Coding Theorem

The source coding theorem says you can represent data at entropy. Shannon’s channel coding theorem says something that initially seems impossible:

For any channel with capacity C and any rate R < C, there exists a code that achieves arbitrarily low error probability. For any rate R > C, the error probability is bounded away from zero regardless of the code used.

This is counterintuitive. Channels have noise; intuitively you might expect some irreducible error rate that you trade off against throughput. Shannon proved that below capacity, you can have arbitrarily low error and a fixed positive rate simultaneously, with a good enough code. Above capacity, errors are unavoidable.

The proof is existential — Shannon showed that a randomly chosen code achieves this, without specifying what code to use. Finding the explicit codes was left to subsequent decades of coding theory. Turbo codes (1993) and LDPC codes (re-discovered in the 1990s, originally Gallager 1962) finally got within a fraction of a decibel of the Shannon limit. Polar codes, proposed by Arikan in 2009, are the first provably capacity-achieving codes with efficient encoding and decoding; they are used in 5G NR.

The channel coding theorem explains the structure of every modern communications system. LDPC codes are inside your Wi-Fi chipset, your LTE modem, your Blu-ray drive, and your satellite receiver. Reed-Solomon codes, developed from the same theoretical tradition, protect data on every hard drive and QR code. The theorem told engineers that perfect reliability was achievable in principle; coding theorists spent fifty years figuring out how to achieve it efficiently.

How Shannon Bounds Real Systems

It is worth being concrete about how entropy and channel capacity constrain the tools that software engineers use daily.

Lossless compression (gzip, zstd, bzip2, xz): bounded by source entropy. For English text at ~1 bit/character of true entropy, the theoretical ceiling is about 8:1 compression. Practical compressors with finite-context models typically achieve 3:1 to 5:1 on typical source code and prose.

Lossy compression (JPEG, MP3, AAC, H.264, AV1): not directly bounded by Shannon entropy in the same way, but bounded by rate-distortion theory, which Shannon also developed. For a given distortion budget (how much degradation you accept), there is a minimum bit rate. Codec engineers spend careers approaching rate-distortion limits with increasingly sophisticated transform coding, quantization, and perceptual models.

Storage media: hard drive and SSD manufacturers use LDPC codes to detect and correct bit errors from the storage medium. The error-correcting headroom they must maintain is set by the channel coding theorem applied to the storage channel (the physical medium as channel, with its own SNR and capacity).

Network protocols: TCP’s throughput is bounded above by the Shannon limit of the physical link. Practical throughput is lower due to protocol overhead, congestion control, and latency, but the ceiling is always Shannon’s.

Cryptographic hash functions and PRNGs: a key desideratum is that their output have entropy approaching 8 bits/byte. Measuring the entropy of PRNG output is a practical security test.

Entropy in Machine Learning: Cross-Entropy Loss

Shannon entropy re-enters the picture in an unexpected place: the training of neural networks, including language models.

The cross-entropy between a true distribution P and a model’s predicted distribution Q is:

H(P, Q) = -sum over all x of P(x) * log2(Q(x))

Cross-entropy is the expected number of bits needed to encode events from P using the code optimized for Q. When your model is perfect — Q = P — cross-entropy equals Shannon entropy H(P), the minimum. When the model is wrong, the extra bits needed are the KL divergence D_KL(P || Q):

H(P, Q) = H(P) + D_KL(P || Q)

Training a language model by minimizing cross-entropy loss is mathematically equivalent to minimizing the KL divergence between the true data distribution and the model’s predicted distribution. Shannon entropy is the floor — the irreducible uncertainty in the data — and the KL term measures how much the model’s distribution diverges from reality.

Perplexity, the standard evaluation metric for language models, is just:

perplexity = 2^H(P, Q)

A perplexity of 10 means the model is, on average, as confused as it would be choosing uniformly among 10 options. Lower perplexity means the model’s probability distribution more closely matches the true distribution. Shannon’s entropy formula from 1948 is embedded in every gradient descent update of every language model trained today.

This is not a coincidence of notation. The same insight — quantify surprise, minimize average surprise — runs from Shannon’s noisy telegraph channel to transformer training. Information theory provides the unified language.

The Kelly Connection

Shannon’s colleague at Bell Labs, John Kelly, extended the entropy framework to a different domain in 1956. Kelly’s paper “A New Interpretation of Information Rate” posed the problem of a gambler receiving tips over a noisy channel and asked how much to bet on each race. His answer: bet a fraction of your bankroll equal to the channel capacity of the information channel between the tipster and the gambler.

For a binary channel with edge e (your probability advantage over a fair bet):

Kelly fraction f* = 2p - 1 = edge

Where p is the probability of winning. The connection to entropy is explicit: the optimal growth rate of the gambler’s wealth equals the channel capacity, which is one minus the binary entropy of the crossover probability. The gambler’s edge is precisely the information advantage the private channel provides over the public market.

Growth rate per bet = C = 1 - H(p_error)

This means the Kelly criterion is not a financial heuristic — it is applied information theory. The optimal bet size is determined by the same formula that determines how fast you can push bits through a noisy wire. The Kelly criterion post on this blog develops the full derivation and the practical complications (parameter estimation error, fractional Kelly, multi-asset generalization), but the fundamental insight belongs to Shannon’s framework. What looks like a finance problem is a channel capacity problem with money as the symbol alphabet.

Verdict

Shannon’s 1948 paper is one of the few theoretical works in engineering where the foundational results remain unbreached. The Shannon-Hartley formula sets limits that no modulation scheme or antenna design can cross; 5G sits closer to those limits than 1G did, but it sits below them. The source coding theorem sets compression floors that no algorithm can undercut; modern compressors are impressively close to those floors on structured data. The channel coding theorem promised reliable communication below capacity and delivered half a century of coding theory that made good on that promise.

The practical payoffs are real: Shannon’s framework tells you why your modem saturated at 56k and why your Wi-Fi link rate is what it is. It tells you to compress before encrypting. It tells you why a language model’s loss function is measuring surprise in Shannon’s sense. It tells you why the Kelly criterion for bet sizing is a channel capacity problem in disguise.

The honest limitation is that Shannon’s theorems are asymptotic — they describe what is achievable with sufficiently long codes and large block lengths. Real systems operate with finite block lengths, finite latency budgets, and imperfect channel estimation. The gap between the theorem and the implementation is where most engineering happens. But the theorem tells you how wide that gap is and whether you are closing it or just adding complexity without gaining information-theoretic ground.

Every engineer who touches compression, wireless, storage, or machine learning is working in a space Shannon defined. The 1948 paper is worth reading in the original.

Sources

Shannon, C. E. (1948). “A Mathematical Theory of Communication.” Bell System Technical Journal, 27(3), 379–423.
Shannon, C. E., and Weaver, W. (1949). The Mathematical Theory of Communication. University of Illinois Press.
Cover, T. M., and Thomas, J. A. (2006). Elements of Information Theory, 2nd ed. Wiley-Interscience.
Gallager, R. G. (1968). Information Theory and Reliable Communication. Wiley.
Kelly, J. L. (1956). “A New Interpretation of Information Rate.” Bell System Technical Journal, 35(4), 917–926.
Arikan, E. (2009). “Channel Polarization: A Method for Constructing Capacity-Achieving Codes for Symmetric Binary-Input Memoryless Channels.” IEEE Transactions on Information Theory, 55(7), 3051–3073.
IEEE 802.11ac specification and 802.11ax specification (Wi-Fi Alliance).
Blahut, R. E. (2010). Algebraic Codes for Data Transmission. Cambridge University Press.