The limits of my language mean the limits of my world. — Wittgenstein From Bytes to BPE: A From-Scratch Tour of LLM Tokenization I wanted to actually trace what happens when I type Hello, world! and hit send. Not the high-level "the model processes your input", but the actual chain of transformations from string to bits to integers. So I opened a Python REPL, imported tiktoken, and worked backward from the integers the model sees to the bytes I'd typed. What I found is that tokenization is doing more work than I'd given it credit for, and that a surprising amount of LLM weirdness makes sense once you see the byte-level mechanics underneath. So what is a token? A token is a chunk of text that has been assigned an integer ID in the model’s vocabulary. The vocabulary is a fixed list, usually somewhere between 30,000 and 200,000 entries, and it is the only thing the model ever actually sees. When I send the model Hello, world!, it doesn't see characters. It sees something like [15496, 11, 995, 0]. Four integers. (The exact IDs depend on which tokenizer, this is roughly what GPT-2's tokenizer produces.) Everything the model knows about language, it learned through statistical relationships between integers in lists like this one. The interesting question is: how do we get from a string of bytes that humans can read to a list of integers that the model finds useful? The answer turns out to be a clever, slightly hacky compression algorithm. From string to bits Computers don’t store strings, they store numbers, and the numbers themselves are stored as bits. So before any tokenization happens, the string has to become a sequence of 0s and 1s. The standard way to do this is UTF-8 encoding . The letter h has Unicode codepoint 104, and 104 in binary is 01101000. UTF-8 uses 1 byte for ASCII characters and up to 4 bytes for everything outside ASCII (Chinese, Arabic, emoji). So hello becomes 40 bits: 01101000 01100101 01101100 01101100 01101111 h e l l o That’s the rawest representation. About as low-level as it gets. From bits to bytes Bits at this granularity are too fine-grained to be useful, so we group them into bytes (8 bits each). Now hello is a sequence of integers in the range 0 to 255: [104, 101, 108, 108, 111] You could just feed these straight into a model. Some byte-level models ( ByT5 , MambaByte ) actually do exactly this. The problem is sequence length. A 5,000-character article becomes ~5,000 byte inputs, and transformer attention is quadratic in sequence length, so you pay roughly 16x more compute than if you could compress those bytes down by a factor of 4 first. (More on that 4x ratio below.) So the question becomes: can we losslessly compress a sequence of bytes into something shorter, in a way that is helpful for a downstream language model? The key observation: byte pairs are wildly non-uniform If you take a large corpus, say all of Wikipedia, and count which adjacent byte pairs show up most often, the distribution is brutal. The pair (116, 104), which is t followed by h, shows up constantly. The pair (101, 32), which is e followed by a space, shows up constantly. The pair (113, 122), which is q followed by z, shows up almost never. This is just the structure of language bleeding into the byte stream. English has a lot of thes, a lot of ings, a lot of vowels followed by spaces. Those byte pairs sit next to each other over and over. So here’s the trick. What if we replaced the most common pair with a single new symbol? That is the entire idea behind Byte Pair Encoding (BPE), which is the algorithm essentially every modern LLM uses (GPT, Claude, Llama, all of them). It is not new; it was originally a data compression technique from the 90s. Somewhere along the way someone realized it also makes a great way to define an LLM’s vocabulary, and now we’re all using it. Building tokens with BPE The algorithm is almost embarrassingly simple: Start with the text as raw bytes (256 possible values). Count every adjacent pair of bytes. Find the most common pair. Say it’s (116, 104), that is th. Add a new symbol to the vocabulary, ex: token 256, and replace every occurrence of (116, 104) with 256. Repeat. Now token 256 (th) frequently sits next to byte 101 (e), forming the. Merge those into token 257. Keep going until the vocabulary is whatever size you wanted (typically 50k to 200k). Each merge is one entry in the vocabulary. After tens of thousands of iterations, you end up with tokens for common whole words (the, and, because), tokens for common subwords (ing, tion, able), tokens for code patterns (def, ==, four spaces of indentation), and crucially, all 256 raw bytes still in there as a fallback for anything weird (emoji, foreign scripts, base64 blobs, typos, novel proper nouns). The whole thing is deterministic and lossless. You can always decode the integers back to the exact original byte sequence. BPE is basically a learned dictionary compression where the dictionary is induced from the statistics of the training data, which (I suspect) is why it works so well: the most efficient encoding tends to be the one that matches the actual frequency structure of the input. Why bother with all this? The natural question: why not just use words, or just use characters? Words don’t work because there are too many of them. English alone has hundreds of thousands of word forms once you count plurals, conjugations, proper nouns, typos, hashtags, slang, and made-up terms. Multiply that across languages and the vocabulary balloons into the millions. And you’d still be stuck the moment a user types something you’ve never seen. Characters don’t work well either , because sequences get really long. “Tokenization is fascinating” is 27 characters but only ~3 tokens. With characters, the model has to spend capacity learning that t-h-e means the, capacity that could have gone to something more interesting. And, again, quadratic attention. Tokens are the compromise. They give you: Roughly 4 characters per token in English, so a 1,000-word article becomes ~1,300 tokens instead of ~5,000 characters. That is about a 4x compression, which translates to ~16x less attention compute at the same length. Universal coverage. Because tokens are built bottom-up from bytes, the model can always fall back to bytes for weird inputs. There is no “unknown word” problem in modern LLMs, which (if you remember the bad old days of NLP) is actually kind of remarkable. Useful units of meaning. tokenization decomposes into token + ization, and the model genuinely benefits from seeing the morphological structure even though no one told BPE about morphology. The weird parts This is also where a lot of LLM weirdness comes from. Most of these I learned by getting bitten by them. The leading space matters. hello and hello (with a leading space) are different token IDs. Two unrelated rows in the embedding table. This is why prompts sometimes need to be formatted in slightly fussy ways, and why a model can behave subtly differently depending on whether a word starts a sentence or sits mid-stream. Numbers tokenize badly. 127 might be one token. 128 might be three. Most likely this is because 127 happened to be common enough in the training data to get its own merge, while 128 didn't. This is part of why early LLMs were terrible at arithmetic; they weren't even seeing digits consistently. Some recent models partially fix this by forcing each digit to be its own token, which helps a lot. Different languages get unequal treatment. English compresses well at ~4 chars/token because English dominates the training corpora. Many other languages take 2 to 3 times more tokens for the same content, which makes them slower and more expensive to use. (This is a real and underdiscussed source of inequality in LLM access.) The tokenizer is model-specific. GPT-4 uses cl100k_base. Llama 3 went to a 128k vocabulary. Claude has its own. The same string can be a wildly different number of tokens depending on which model you're talking to, which is annoying when you're trying to estimate costs or context budgets across providers. Reflections The thing that surprised me most going through this exercise is how empirical tokenization is. There is no theory of language baked in. BPE doesn’t know what a morpheme is, doesn’t know that ing is a suffix, doesn't know the is a determiner. It just counts byte pairs. The fact that the resulting tokens look so suspiciously linguistic (whole words, common suffixes, sensible subword splits) is purely a downstream consequence of frequency statistics in the training corpus. Language structure leaks through because language is statistical, and BPE is a faithful mirror. It also makes me think the tokenizer is doing more representational work than people give it credit for. The model has to learn to operate on tokens, but the tokens themselves were chosen by an algorithm that already encoded a lot of structure (frequency, locality, morphology-as-side-effect) into the input space. You can’t fully separate “what the model knows” from “what the tokenizer set up for it.” Looking forward I suspect tokenization as we know it will look dated within 5 years. There is a steady drumbeat of research into byte-level and character-level models (Mamba variants, ByT5, Charformer), and the main blocker has always been quadratic attention cost. As architectures get cheaper at long sequences (linear attention, state-space models, hierarchical attention), the case for BPE weakens. The “leading space matters” footgun and the multilingual unfairness both just go away if you process bytes directly. But (and this is the hedge) BPE has been shockingly resilient. Every “tokenizer-free” approach so far has had subtle disadvantages in either compute or quality, and the field keeps coming back to BPE. So maybe in 5 years we’re still here, just with bigger vocabularies and slightly better merge rules. ¯\ (ツ) /¯ If you want to play with this directly, the cleanest implementation I have come across is minbpe, which trains a BPE tokenizer in about 100 lines of Python. Reading it and tracing the merges by hand is, by a wide margin, the fastest way to actually internalize what tokens are. From Bytes to BPE: A From-Scratch Tour of LLM Tokenization was originally published in GoPenAI on Medium, where people are continuing the conversation by highlighting and responding to this story.