POST-TRAINING & DISTILLATION

MODULE 2: DATA ENGINEERING & SYNTHETIC DATA · SCENE 05

Embedding Clustering & Deduplication

MinHash dedup and embedding-space clustering that keeps topic coverage balanced across the post-training corpus.

19,600 PROMPTS
coding4,200math3,100chat9,800safety600 · UNDER-FEDcreative1,500 · UNDER-FEDtools400 · UNDER-FED

One cluster at 45% of the corpus is a diversity bug — downsample it, feed the thin leaves.

ARCHITECTURE & FORMULATIONModule 2: Data Engineering & Synthetic Data

Embedding diversity and MinHash deduplication

Embedding-space clustering turns an unstructured pile of examples into a measurable distribution. MinHash-LSH removes near-duplicates cheaply; cluster counts expose which capability regions are over- and under-represented.

MinHash + LSH

Shingle each document, hash it into a signature, bucket via locality-sensitive hashing. Near-duplicates collide without an O(n²) all-pairs comparison.

Cluster Coverage

Embed prompts, cluster with k-means or HDBSCAN, then inspect cluster sizes. A single cluster holding 40% of the data is a diversity bug.

Deliberate Rebalancing

Downsample bloated clusters, synthesize examples for empty ones. Coverage is a design decision.

MATHEMATICAL FORMULATION · SIGNATURE SIMILARITY
J(A, B) ≈ agree(MinHash_sig_A, MinHash_sig_B) / signature_length

The fraction of matching min-hashes estimates Jaccard similarity between document shingle sets; LSH turns that estimate into cheap candidate pairs.

REAL-WORLD PRODUCTION ENGINEERING
  • The FineWeb and Dolma pipelines dedup trillions of tokens with MinHash-LSH before any model sees the data.
  • Teams keep a 'cluster dashboard' during SFT data review — removing one over-represented cluster often fixes a specific behavioral tic.
← 04 · Data Curation & Quality Filtering
06 · Self-Instruct, Evol-Instruct & Rejection Sampling →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]