Language models never see text. They see integers. The layer that turns a string into a list of token IDs is byte-pair encoding, a 1994 compression trick repurposed to carve words into subword units. It quietly decides your context window, your bill, and why the model cannot spell.
Transformers
-
Tokenization and Byte-Pair Encoding: How Text Becomes Numbers