Results dashboard

Generated by uv run adapterops dashboard from committed runs — no number here is typed by hand. Six metrics (PRD §11). The two quality rows are separate tables and are never averaged together (F32).

Sources: runs/regression__v1-baseline-1.json · runs/regression__v1-baseline-2.json · evals/GATE_THRESHOLDS.json · runs/m1_serving.json · runs/economics.json · runs/router__operating_curve__judged.json · runs/router__v2.json · runs/router__rules.json · runs/judge__cost.json · runs/frontier__golden.json · runs/drafting__m2_gpt4o__frontier.json · runs/drafting__m2_gpt4o.json · runs/regression__intent-shuffled.json · runs/regression__all-m11.json · manifests/system.json

1 · Quality retained — random golden set

Two runs of the unchanged v1 manifest. A threshold is 3× the larger of inference and training spread (D37); only intent has a measured training spread, so only intent is enforced.

task	gated metric	baseline run 1	baseline run 2	spread	threshold	gate
intent	micro_accuracy (n=770)	0.9286	0.9299	0.0013	0.0117	enforced (training variance)
urgency	macro_f1 (n=300)	0.4096	0.4223	0.0127	0.0381	provisional, not enforced
pii	span_f1_strict (n=300)	0.9462	0.9453	0.0009	0.0027	provisional, not enforced
drafting	judge_score_mean (n=300)	4.2709	4.2844	0.0135	0.0405	provisional, not enforced

Frontier reference, not a gate. GPT-4o-mini with the escalation arm's prompts, on the same golden items. Drafting is compared on GPT-4o grades, which also grade GPT-4o-mini.

task	this adapter	GPT-4o-mini
intent (micro_accuracy)	0.9286	0.6870
urgency (macro_f1)	0.4096	0.3824
pii (span_f1_strict)	0.9462	0.6663
drafting (GPT-4o grade)	4.2367	4.5167

2 · Quality retained — hard cases

Mined from v1's failures and adjudicated for label noise. Scores near 0 are expected by construction, which is why this split cannot gate — see M11 below.

task	gated metric	baseline run 1	baseline run 2	spread	threshold	gate
intent	micro_accuracy (n=57)	0.0000	0.0000	0.0000	—	report-only — floor 0, no threshold
urgency	macro_f1 (n=150)	0.0068	0.0068	0.0000	—	report-only — floor 0, no threshold
pii	span_f1_strict (n=150)	0.8520	0.8524	0.0004	0.0012	provisional, not enforced
drafting	judge_score_mean (n=113)	3.6918	3.7290	0.0372	0.1116	provisional, not enforced

3 · Frontier-call rate

Not measured live — no router runs in the serving path. What exists is the offline operating curve over the router pool, read at a 20% escalation budget. Drafting is graded by GPT-4o on both arms.

population	never escalate	GPT-4o-mini alone	policy	escalated	quality	share of available gain
router_in_distribution	0.727	0.523	confidence	19.9%	0.781	57%
router_in_distribution	0.727	0.523	router_p_fail	19.9%	0.709	-19%
router_in_distribution	0.727	0.523	random	19.9%	0.702	-27%
router_shift	0.718	0.513	confidence	20.0%	0.762	43%
router_shift	0.718	0.513	router_p_fail	20.0%	0.687	-31%
router_shift	0.718	0.513	random	20.0%	0.681	-36%

Pre-registered retry (D42). Three fixes for the learned router, frozen before training. Quality difference from confidence at a 20% budget, with 95% paired bootstrap intervals:

variant	in-distribution	shift	verdict
confidence_per_task	-0.043 [-0.069, -0.013]	-0.030 [-0.045, -0.011]	loses to confidence
cascade_lr	-0.041 [-0.066, -0.015]	-0.024 [-0.041, -0.006]	loses to confidence
gain_text_s11	-0.046 [-0.071, -0.015]	-0.043 [-0.060, -0.025]	loses to confidence
gain_text_s22	-0.066 [-0.100, -0.036]	-0.056 [-0.078, -0.036]	loses to confidence
gain_text_s33	-0.074 [-0.107, -0.041]	-0.070 [-0.092, -0.047]	loses to confidence

Rules-based baseline (F26), reference only. Written after the D42 allocation table was read; the task order comes from the train split. At a 20% budget:

rule	gain captured (in-dist / shift)	vs confidence, in-dist	vs confidence, shift
rules_length	-54% / -62%	-0.105 [-0.138, -0.074]	-0.107 [-0.127, -0.085]
rules_task_length	38% / 37%	-0.018 [-0.033, -0.003]	-0.006 [-0.016, +0.008]

The adapter's own confidence is the only policy that pays. The learned router is worse than never escalating, and GPT-4o-mini on its own scores below the local adapters.

4 · Cost per 1K requests — derived, not billed

	value
GPT-4o-mini, same 5,800 pairs, list price	$0.0572
local A10 at M1 throughput (23.6 rps), $0.75/h assumed	$0.0088
break-even sustained load	3.64 req/s (13,112 req/h)
ratio, only if the GPU is never idle	6.5×
weights on disk: one base + 4 adapters vs a full copy per task	3.38 GB vs 12.35 GB
judging 1K drafting replies: GPT-4o, token-derived over 2,800 grades	$1.30
judging 1K drafting replies: distilled judge, Darwin arm64 CPU, no API	59 s

Not measured: gpu memory at serving (no nvidia-smi or vLLM memory figure was recorded); frontier latency (the F8 run recorded tokens, not timings); max local throughput (no saturation sweep was run).

5 · P95 latency — M1, A10, four adapters at once, concurrency 16

adapter	requests	P50 ms	P95 ms	max new tokens	mean generated
drafting	1,450	1,249	3,000	448	123.6
intent	1,450	82	136	12	5.0
pii	1,450	858	2,259	384	80.4
urgency	1,450	51	60	6	2.0

6 · Fallback rate

0.0% — 0 errors in 5,800 requests during M1. Measured once, on that run; the regression runs did not record it.

Failure demo

M7 — detect → block → rollback, on real serving

A shuffled-label intent adapter (F21) was served as intent and scored against the last known-good manifest.

step	record
detect	intent micro-accuracy 0.0052 vs baseline 0.9286 — drop 0.9234 against an enforced threshold of 0.0117
block → force	promotion refused for: `intent: regression 0.9234 exceeds threshold 0.0117` — forced as v3 with the override recorded in the manifest
rollback	current manifest is v2, rolled back from v3

Manifest history:

version	note	gate	promoted despite
v1	Phase 2 baseline: four adapters, splits frozen	report_only	—
v2	Re-freeze: pin the D38 hard split the Phase 4 baselines were measured on, the router checkpoint, and the F33 gate	enforcing	—
v3	forced, to prove rollback	enforcing	intent: regression 0.9234 exceeds threshold 0.0117

M11 — does the hard split catch what the random set misses?

All four under-trained checkpoints (15% of training steps) scored on both splits against the v1 baseline. A positive drop is a regression.

task	random drop	random threshold	hard drop	hard threshold
intent	0.2026	0.0117 (enforced (training variance))	-0.1579	— (report-only — floor 0, no threshold)
urgency	0.0874	0.0381 (provisional, not enforced)	-0.0812	— (report-only — floor 0, no threshold)
pii	0.2245	0.0027 (provisional, not enforced)	0.2154	0.0012 (provisional, not enforced)
drafting	0.1508	0.0405 (provisional, not enforced)	-0.2700	0.1116 (provisional, not enforced)

Answer: no. The random set flags every checkpoint; the hard split flags only PII and improves on the others. The hard split was mined from the v1 adapter's own failures, so v1 scores 0.0000 on intent's hard cases and 0.0068 on urgency's by construction. Any model whose errors differ from v1's scores higher there. A split mined from one model's failures measures difference from that model, not difficulty — it stays report-only.

Four adapters, one base model, the negative results kept