Tokenizer Design & Byte-Pair Encoding (BPE)
Tokenizers convert raw text strings into discrete integer sequences for model ingestion. Modern BPE tokenizers operate directly on UTF-8 bytes to guarantee zero out-of-vocabulary (OOV) tokens while maximizing compression efficiency.
Byte-Level BPE
Starting with a base vocabulary of 256 raw UTF-8 bytes, the algorithm iteratively counts adjacent symbol pairs across a massive text corpus and merges the most frequent pair into a new token.
Vocabulary Size Trade-offs
Small vocabs (32k) save memory in the embedding/LM head layer but result in longer sequence lengths per document. Large vocabs (128k+) shorten token sequences by ~15%-20%, speeding up attention and generation.
Pre-Tokenization Regex
Regex splitting (such as tiktoken's cl100k pattern) prevents BPE from merging across numbers, punctuation, or word boundaries, preventing weird token mergers.
Find the most frequent pair of adjacent tokens in the corpus and assign it a fresh vocabulary index.
- Llama 3 expanded vocabulary from Llama 2's 32k to 128k tokens, achieving ~15% better text compression and significantly improving non-English performance.