EVALUATION

MODULE 1: EVAL FOUNDATIONS · SCENE 02

Capability vs Alignment Evals

Can it do the task, and does it do it the way we intended? Two axes that fail independently and need separate instruments.

CAPABILITY0.85ALIGNMENT0.65
BALANCED DIAGONALCAPABILITY →ALIGNMENTMODELMEAN 0.75 · GAP 0.20 · AXES DIVERGE — DO NOT SHIP ON THE MEAN

A mean of 0.75 hides a gap of 0.20. Capability can soar while alignment collapses in the same fine-tune.

ARCHITECTURE & FORMULATIONModule 1: Eval Foundations

Capability and alignment are separate axes

Capability evals ask 'can the model do the task?' — math, code, retrieval, reasoning. Alignment evals ask 'does it do the task the way we intended?' — instruction-following, tone, refusal behavior, honesty. They fail independently, so a single aggregate score hides the axis that broke.

Knowledge and skill

Capability instruments are usually graded against a verifiable reference: a checked answer, compiling code, a passing test.

Intent match

Alignment instruments grade against a policy or preference: did it follow the format, respect a boundary, avoid over-refusing?

Independent failure

A fine-tune can raise capability while collapsing safety behavior. Report both axes or you will ship half a system.

MATHEMATICAL FORMULATION · TWIN-AXIS REPORTING
score = (capability + alignment) / 2, gap = |capability − alignment|

A composite of 0.80 can hide (0.95 capability, 0.65 alignment) or (0.80, 0.80). Always publish the pair and the gap, not just the mean.

REAL-WORLD PRODUCTION ENGINEERING
  • Frontier model cards report capability suites (MMLU, GPQA, SWE-bench) and safety suites (red-team, refusal, over-refusal) separately.
  • A capability win that trips a safety gate is a release blocker regardless of the average.
← 01 · The Evaluation Loop
03 · Static Suites vs Dynamic Arenas →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]