Final test results and failure analysis
Published: 2026-08-08
This report is the final status of the evidence retained in this repository.
PASS means the cited evidence satisfies the experiment's stated acceptance
criterion. FAIL means an attempted execution did not satisfy it.
INCONCLUSIVE means the retained evidence cannot support the intended claim.
The two local contract tests are reported separately from broker-backed
experiments.
Results
| Scope | Status | Observed result | Evidence |
|---|---|---|---|
| Python contract tests | PASS | 2 passed | pytest -q on 2026-08-08 |
| 01 — Multiple AI | PASS | One work event produced three independent proposals. | experiments/01-multiple-ai/results/run-001.md |
| 02 — Human review | PASS | A human participant emitted ProposalApproved. |
experiments/02-human-review/results/run-001.md |
| 03 — Event chain | PASS | An OCI-broker four-event causal chain completed. | experiments/03-event-chain/evidence/run-002/ |
| 04 — Event replay | PASS | A fresh replay reconstructed state from all recorded event IDs. | experiments/04-event-replay/evidence/run-003/ |
| 05 — Failure recovery | INCONCLUSIVE | Two WorkCreated events yielded only one ProposalCreated. |
experiments/05-failure-recovery/RESULT.md |
| 06 — Live LLM | PASS | A real-broker Luna participant produced the required live ProposalCreated event. |
experiments/06-live-llm/evidence/run-005/ |
| 07 — Provider hot swap | PASS | Deterministic and Luna participants produced proposals at one event contract. | experiments/07-provider-hot-swap/evidence/run-002/ |
| 08 — Competing consumers | INCONCLUSIVE | No worker output or benchmark measurement was retained. | experiments/08-parallel-scaling/RESULT.md |
| 09 — Audit trail | PASS | OCI-broker lineage generated a Mermaid audit graph. | experiments/09-audit-trail/evidence/run-002/ |
| 10 — End-to-end SDLC | PASS | The corrected OCI-broker run recorded ten events ending in DocumentationCreated. |
experiments/10-end-to-end-sdlc/evidence/run-003/ |
Failures and their causes
05 — Failure recovery: inconclusive
The expected recovery result was two proposals for two retained work inputs; only one proposal was recorded. The existing result identifies the likely mechanism as a consumer readiness/partition-assignment race around the worker restart. That mechanism is not proven: raw worker logs, offsets, and assignment telemetry were not retained. The defensible conclusion is that retention alone did not demonstrate reliable recovery in this run.
06 — Live LLM: historical configuration/provider failures
All three retained attempts ended before a proposal event was written:
| Run | Direct error evidence | Cause supported by evidence |
|---|---|---|
run-001 |
Azure-compatible request returned HTTP 404 DeploymentNotFound. |
The configured deployment did not exist or was not available to the endpoint at execution time. |
run-002 |
Request returned HTTP 400: invalid configured model identifier. |
The configured model identifier was invalid for that provider. |
run-003 |
Request returned HTTP 404: configured function not found for the account. | The configured provider target was unavailable to that account. |
These are provider configuration/availability failures, not model-quality
findings. They were superseded by successful real-broker run run-005, which
used Azure Sweden and gpt-5.6-luna. See each run's
requirements-1.stderr.log and failure.json.
07 — Provider hot swap: historical failed live stage
The live participant in run-001 returned HTTP 404 DeploymentNotFound.
The configured Azure-compatible deployment was unavailable at that time, so
the sequence could not complete. This was superseded by run-002: the
Azure Sweden/Luna participant and two deterministic participants each emitted
a proposal from the same input contract. The concurrent run supports
contract-level substitutability, not an ordered runtime hand-off.
08 — Competing consumers: inconclusive
Three worker log files were empty and no measurement JSON was produced. The retained evidence therefore establishes neither partition sharing nor throughput. The precise cause is unknown because worker readiness, consumer assignment/rebalance state, host scheduling, and observer instrumentation were not captured. A readiness/assignment problem is plausible, but is not a proven root cause.
06 — Participant identity consistency defect
The passing run-005 event has authoritative top-level source and
actor_id values of ai.requirements.azure.v1, but its copied
payload.participant_id is ai.requirements.deterministic.v1. The event is
retained unchanged. The experiment passed at the event-contract level while
exposing a participant-identity consistency defect in the payload. The
implementation now writes the live participant ID consistently; Experiment
07's retained live event demonstrates the correction.
10 — First SDLC attempt: harness failure, corrected by retry
The first run failed because the bounded participant raised
RuntimeError: consumer ended before receiving the expected event in
documentation-9. This is a test-harness lifecycle/readiness failure rather
than an SDLC-topology failure: the readiness-checked retry completed and is
the basis for the final PASS. The original failure is retained at
experiments/10-end-to-end-sdlc/evidence/run-001/.
Implications
The passing runs support causal event exchange, replay, audit lineage, the bounded SDLC topology, and one bounded Azure Sweden/Luna participant through the real broker. They do not establish multi-provider live substitution, reliable recovery, or competing-consumer performance. The inconclusive runs show that a Kafka-compatible event log supplies durable transport but not worker lifecycle coordination, assignment visibility, or execution accounting.