TRAINING AT SCALE

MODULE 2: DATA AT WEB SCALE · SCENE 08

Building the vocabulary

Byte-pair encoding grows a vocabulary from raw bytes — merge by merge, with rules that shape every token.

RAW TEXT: "thinking"VOCAB TOKENS: 8
"t""h""i""n""k""i""n""g"

Byte-Pair Encoding merges frequent adjacent character pairs into single vocabulary tokens.

ARCHITECTURE & FORMULATIONModule 2: Data Engineering & Curation at Web-Scale

Tokenizer Design & Byte-Pair Encoding (BPE)

Tokenizers convert raw text strings into discrete integer sequences for model ingestion. Modern BPE tokenizers operate directly on UTF-8 bytes to guarantee zero out-of-vocabulary (OOV) tokens while maximizing compression efficiency.

Byte-Level BPE

Starting with a base vocabulary of 256 raw UTF-8 bytes, the algorithm iteratively counts adjacent symbol pairs across a massive text corpus and merges the most frequent pair into a new token.

Vocabulary Size Trade-offs

Small vocabs (32k) save memory in the embedding/LM head layer but result in longer sequence lengths per document. Large vocabs (128k+) shorten token sequences by ~15%-20%, speeding up attention and generation.

Pre-Tokenization Regex

Regex splitting (such as tiktoken's cl100k pattern) prevents BPE from merging across numbers, punctuation, or word boundaries, preventing weird token mergers.

MATHEMATICAL FORMULATION · BPE PAIR FREQUENCY MERGE
Pair_selected = argmax_(p_A, p_B) Count( p_A, p_B )

Find the most frequent pair of adjacent tokens in the corpus and assign it a fresh vocabulary index.

REAL-WORLD PRODUCTION ENGINEERING
  • Llama 3 expanded vocabulary from Llama 2's 32k to 128k tokens, achieving ~15% better text compression and significantly improving non-English performance.
← 07 · Delete first, train later
09 · Spend tokens like money →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]