AI ENGVisual Encyclopedia

MODULE 2: DATA ENGINEERING & SYNTHETIC DATA · SCENE 05

Embedding Clustering & Deduplication

MinHash dedup and embedding-space clustering that keeps topic coverage balanced across the post-training corpus.

19,600 PROMPTS
coding4,200math3,100chat9,800safety600 · UNDER-FEDcreative1,500 · UNDER-FEDtools400 · UNDER-FED

One cluster at 45% of the corpus is a diversity bug — downsample it, feed the thin leaves.

TECHNICAL BREAKDOWNModule 2: Data Engineering & Synthetic Data

Embedding diversity and MinHash deduplication

Embedding-space clustering turns an unstructured pile of examples into a measurable distribution. MinHash-LSH removes near-duplicates cheaply; cluster counts expose which capability regions are over- and under-represented.

MinHash + LSH

Shingle each document, hash it into a signature, bucket via locality-sensitive hashing. Near-duplicates collide without an O(n²) all-pairs comparison.

Cluster Coverage

Embed prompts, cluster with k-means or HDBSCAN, then inspect cluster sizes. A single cluster holding 40% of the data is a diversity bug.

Deliberate Rebalancing

Downsample bloated clusters, synthesize examples for empty ones. Coverage is a design decision.

MATHEMATICAL FORMULATION · SIGNATURE SIMILARITY
J(A, B) ≈ agree(MinHash_sig_A, MinHash_sig_B) / signature_length

The fraction of matching min-hashes estimates Jaccard similarity between document shingle sets; LSH turns that estimate into cheap candidate pairs.

REAL-WORLD PRODUCTION ENGINEERING
  • The FineWeb and Dolma pipelines dedup trillions of tokens with MinHash-LSH before any model sees the data.
  • Teams keep a 'cluster dashboard' during SFT data review — removing one over-represented cluster often fixes a specific behavioral tic.
← 04 · Data Curation & Quality Filtering
06 · Self-Instruct, Evol-Instruct & Rejection Sampling →