Capability and alignment are separate axes
Capability evals ask 'can the model do the task?' — math, code, retrieval, reasoning. Alignment evals ask 'does it do the task the way we intended?' — instruction-following, tone, refusal behavior, honesty. They fail independently, so a single aggregate score hides the axis that broke.
Knowledge and skill
Capability instruments are usually graded against a verifiable reference: a checked answer, compiling code, a passing test.
Intent match
Alignment instruments grade against a policy or preference: did it follow the format, respect a boundary, avoid over-refusing?
Independent failure
A fine-tune can raise capability while collapsing safety behavior. Report both axes or you will ship half a system.
A composite of 0.80 can hide (0.95 capability, 0.65 alignment) or (0.80, 0.80). Always publish the pair and the gap, not just the mean.
- Frontier model cards report capability suites (MMLU, GPQA, SWE-bench) and safety suites (red-team, refusal, over-refusal) separately.
- A capability win that trips a safety gate is a release blocker regardless of the average.