POST-TRAINING & DISTILLATION

MODULE 2: DATA ENGINEERING & SYNTHETIC DATA · SCENE 06

Self-Instruct, Evol-Instruct & Rejection Sampling

Synthetic data generation loops, model-collapse risk, and the quality gate that rejects low-fidelity generations.

ROUND · 0/530% REPLAY · 93% RETAINEDPURE SELF-TRAIN · 90% RETAINED
1. SEED PROMPTS→2. GENERATE→3. FILTER→4. ADD TO DATAR0R1R2R3R4R5■ REPLAY MIX■ PURE SYNTHETIC (COLLAPSE)

Each round: seed prompts are evolved, answered, filtered, and added back. RUN to watch five rounds.

ARCHITECTURE & FORMULATIONModule 2: Data Engineering & Synthetic Data

Self-Instruct, Evol-Instruct, and model collapse

When human demonstrations are scarce, models generate their own training data: seed prompts are expanded (Self-Instruct) or evolved into harder variants (Evol-Instruct), answered, filtered, and fed back. Without fresh external grounding, repeated self-training degrades diversity — model collapse.

Self-Instruct Loop

Seed prompts → LLM generates new prompts + responses → filter → add to dataset. Bootstrap coverage quickly from ~175 seed tasks.

Evol-Instruct

Each generation rewrites prompts to be more complex, constrained, or multi-step — WizardLM's recipe for escalating difficulty.

Collapse Guard

Mix synthetic data with human/curated data each round, and track output entropy. Pure self-training rounds slowly shrink the distribution's tails.

MATHEMATICAL FORMULATION · RETENTION UNDER REPLAY
retention = 1 − forgetting_rate × (1 − replay_share)

Keeping a fraction of real data in every synthetic round bounds capability loss: a 10% forgetting rate with 30% replay retains ~93% of the original distribution.

REAL-WORLD PRODUCTION ENGINEERING
  • Orca and Phi-style curricula show synthetic data quality depends on reasoning traces, not just answers — generate the rationale, then grade it.
  • Every synthetic round should end at a rejection gate (next scene); unfiltered self-training is how collapse sneaks in.
← 05 · Embedding Clustering & Deduplication
07 · LoRA: Why Low-Rank Adaptation Works →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]