Curation heuristics and the data taxonomy
Post-training datasets are tiny, so every example has outsized influence. Curation combines cheap heuristics (length, symbol ratios, language ID), classifier scoring, and a deliberate taxonomy so capability coverage is designed, not accidental.
Heuristic Filters
Drop examples with degenerate repetition, broken encoding, extreme symbol ratios, or wrong language. These cheap rules remove the majority of low-quality candidates.
Taxonomy Design
Build a tree: instruction following, multi-turn dialogue, code, reasoning, safety, creative writing. Budget counts per leaf so weak leaves are filled rather than duplicated.
Quality over Quantity
LIMA showed ~1,000 excellent examples can beat 50k mediocre ones. Deduplication and per-leaf curation matter more than raw scale.
Near-duplicate prompts teach the model to parrot one phrasing. Keeping unique-prompt share high preserves output diversity at generation time.
- Production pipelines score every candidate with a fast quality classifier and sample the kept set per taxonomy leaf, not globally.
- Safety leaves are budgeted separately — you cannot rely on the general distribution to include enough refusal and boundary examples.