Scoring harness: voxint score

The harness (src/voxint/harness/) is the offline quality-measurement layer: pure, DB-free cores plus file-based CLI adapters. Nothing in it touches settings, the database, or the worker. voxint score … runs on any machine against plain JSON/JSONL files (pip install voxint is all it needs; no Docker stack).

All JSON documents (aliases, enrollment, thresholds) and all output records carry "schema_version": 1; input JSONL streams (name-accuracy items, agreement slots) are versioned by their command's contract rather than per record. Unknown extra fields are ignored; missing/malformed required fields (including non-finite numbers and negative durations) are an error reported with file and line number, exit code 2. Reports are written atomically (temp file + rename) with deterministic key ordering, so identical inputs produce byte-identical outputs. A small synthetic dataset exercising all three commands lives in examples/.

Why these scorers exist

Structural diarization metrics (DER/JER, permutation-optimal WER variants) optimally relabel speakers before scoring, so they are blind to whether the name shown to a user is the right person. The harness measures exactly that:

• score name-accuracy scores assigned display names against ground truth with a strict person-matcher (a bare first name is not proof of identity), and compares two runs with paired statistics.
• score agreement reports one embedding voter's conservative acoustic verdict: is this curated host's voice actually present in this item, judged by cosine against a held-out voiceprint, independent of any LLM or name surface.
• score ensemble fuses two voters' verdicts. Verdicts only: the ensemble layer cannot see vectors, so cross-embedding-space comparison is structurally impossible (see "The cross-space invariant" below).

voxint score name-accuracy

voxint score name-accuracy items.jsonl [--baseline base.jsonl] \
    [--aliases aliases.json] [--target-accuracy 0.95] [--seed 0] [--out report.json]

Input: items JSONL (one object per line)

{"item_id": "ep-001",
 "slots": {
   "SPEAKER_00": {"assigned_name": "Dana Fox", "truth": "Dana Fox",
                  "confidence": 0.91, "duration": 412.5},
   "SPEAKER_01": {"assigned_name": null, "truth": "__ABSTAIN__"}}}

• item_id: unique non-empty string; a duplicate is an error.
• slots: non-empty object of slot label → fields.
• assigned_name: the display name the system produced, or null/a placeholder (speaker_…, auto_…, unknown…) for an abstention.
• truth: a real person name, "__ABSTAIN__" (no name should be assigned), or "__NEITHER_DETERMINABLE__"/null (unscoreable → excluded).
• confidence (optional): feeds the descriptive risk-coverage curve only.
• duration (optional): the slot's weight in the duration-weighted metrics; omitted slots weigh 1.0.

Aliases JSON (optional)

{"schema_version": 1,
 "aliases": {"Daniela Fox": ["Dana Fox", "D. Fox"]}}

Both names appearing under one canonical entry (the key counts as a member) match at alias strength.

Verdicts

Per slot: TP (correct person), FP_WRONG (a different real person), FP_OVERNAME (named where truth is abstain), FN (missed a real person), TN (correct abstain), EXCLUDED (unscoreable). Name matching is Unicode-aware (NFKC + casefold) and strict: id equality > alias table > exact string > surname + given(-initial); single-token containment never matches.

Output report

One JSON object: counts and precision/recall/F1 (plain and duration-weighted), a confusion matrix, slot accuracy with a 95% Wilson CI, per_item verdicts, and, when any confidence was supplied, a descriptive risk_coverage curve (never a gate input: confidence is not proven calibrated).

With --baseline, both files must cover identical item_ids and slot labels and agree on every slot's truth (paired statistics are meaningless across diverging ground truth, so a mismatch is an error); the report gains a paired block: exact McNemar on discordant slot pairs plus an item-clustered bootstrap CI on the mean per-slot delta (deterministic for a given --seed).

voxint score agreement

voxint score agreement --slots slots.jsonl --enrollment enrollment.json \
    --thresholds thresholds.json [--out verdicts.jsonl]

Enrollment JSON: one embedding space per file

{"schema_version": 1, "embedding_space": "acme-voice-v1", "dims": 192,
 "voiceprints": {
   "host-dana": {"embedding": [0.01, "…"], "enrollment_items": 5,
                  "held_out": true, "source_item_ids": ["ep-002", "ep-003"]}}}

• embedding_space: the model/space tag; every vector in this file and in the slots file is bound to it.
• held_out: attests the voiceprint was built only from other items. A false value abstains every use (session_leakage_risk).
• source_item_ids: the items the voiceprint was built from. Scoring an item in this list abstains (session_leakage_risk): a voiceprint must never judge the item it was built from.
• enrollment_items below the thresholds' min_enrollment_items abstains (weak_enrollment).

Thresholds JSON

{"schema_version": 1, "tau": 0.62, "margin": 0.08, "min_duration": 45.0,
 "min_segments": 6, "low_band": 0.35, "neg_min_total_duration": 300.0,
 "min_enrollment_items": 3}

Validated on load: low_band <= tau, cosines within [-1, 1], non-negative floors. Choose values by impostor-trial calibration (voxint.harness.agreement.far_frr_at + one-sided Wilson bounds).

Slots JSONL

{"item_id": "ep-001", "kind": "curated", "host_id": "host-dana",
 "embedding_space": "acme-voice-v1", "total_speech": 1810.0,
 "slots": {"SPEAKER_00": {"embedding": [0.02, "…"], "duration": 412.5,
                           "segments": 44}}}

• kind: curated (score host_id's voiceprint) or negative_control (a no-host channel: score all usable voiceprints, expecting absence).
• embedding_space is required on every record and must equal the enrollment file's. The record proves its space rather than inheriting a tag, so vectors from a different (even equal-dimensional) model are rejected.
• Embeddings are validated (finite, non-zero, dims-length).

Output verdicts JSONL

One object per item: verdict (CONFIDENT_HOST_PRESENT, NO_CURATED_HOST_DETECTED, or ABSTAIN + reason), evidence (host_slot, top_cosine, runner_up_cosine, margin, duration/segments), a contradiction flag (curated host confidently absent on their own channel, or present on a negative control: candidate channel-fact errors for human review), and the embedding_space. Verdicts are silver evidence, never gold truth; the bias is deliberately conservative (abstain on near-ties, short slots, weak/leaking enrollment, low cosine).

voxint score ensemble

voxint score ensemble titanet-verdicts.jsonl other-verdicts.jsonl [--out fused.jsonl]

Joins two agreement outputs on item_id (must cover identical items with matching kinds, and the two files must be in different embedding spaces; two runs of the same model are not independent voters). Records are validated semantically before fusion: verdicts must fit the item kind, a confident-present must carry its winning host_slot, contradiction must be a literal boolean. Then it AND-gates them: both confident on the same slot → SILVER_HOST_PRESENT; both confidently absent (negative controls) → SILVER_NO_HOST; any contradiction, slot mismatch, or single-voter confidence → FLAG_REVIEW; otherwise ABSTAIN. Voter names in reasons are the files' embedding-space tags.

The cross-space invariant

Different embedding models emit vectors in different spaces, and two spaces can share a dimensionality, so a dims check is not an isolation check. The harness enforces isolation structurally:

1. Every vector is a TaggedVector carrying its embedding_space; every cosine entry point (voxint.harness.vectors.cosine) refuses mismatched spaces before touching numpy.
2. One agreement invocation handles exactly one space (the enrollment file defines it; a slots record scored against it inherits it).
3. Voter fusion (voxint.harness.ensemble) accepts only typed verdicts (no numpy import, no vector parameter), so cross-space comparison cannot be expressed at the ensemble layer.

This mirrors the pipeline-side invariant in docs/architecture.md (speaker matching filters by embedding_space); the guardrail tests in tests/unit/test_harness_vectors.py and tests/unit/test_harness_ensemble.py pin it.

Library-only pieces

• voxint.harness.gate_metrics.assemble_gate_metrics: paired baseline/candidate verdict records → release-gate counts (host regressions, correct-to-wrong swaps, over-naming introduction, audited-subset regressions, one-sided-95% Wilson upper bound on the item regression rate). A paired_tally with n_items 0 yields an upper bound of 1.0: "no information" must read as unprovable, not as safe.
• voxint.harness.goldset_strata provides deterministic (SHA-256-spread) priority-ordered stratified sampling plus provenance-gated auto-labeling: a channel fact auto-labels a host truth only when the host's voiceprint is groundable on that item; everything else routes to a human label queue.

These have no CLI yet; if one grows a public workflow, it gets a score subcommand and a contract section here.

Attribution evaluation (issue #113)

Three modules that measure end-to-end speaker attribution accuracy against corpus gold labels (AMI). Pure, DB-free, composable in sequence.

• voxint.harness.ami_recurrence: parses AMI meetings.xml for cross-session speaker recurrence. base_session_id groups scenario suffixes (a/b/c/d) into one day-session. check_kill_criterion applies two explicit viability gates: baseline (>=8 cross-session speakers, >=50 genuine pairs) and calibration (>=50 independent clusters, expected to fail on AMI alone). The recurrence report feeds the protocol manifest.

• voxint.harness.attribution_protocol: protocol manifest dataclasses (AttributionProtocolRow, ProtocolManifest) defining which meetings, speakers, and channels form the evaluation corpus. validate_session_honesty checks that enrollment and test data are base-session-disjoint (fail-closed). JSON round-trip via serialize_manifest / parse_manifest; the speakers detail on RecurrenceReport is intentionally non-persistent.

• voxint.harness.attribution_aligner: the spine. Builds a complete gold-speaker x predicted-slot duration-overlap matrix (with configurable collar, default 0.25s). Classifies each slot: genuine, impostor, mixed, or unscoreable (purity/coverage/margin/eligibility). Purity and coverage are clamped to [0, 1] to handle overlapping same-label intervals. build_trials produces calibration.Trial objects with truth_source="corpus_gold" and cluster_id = gold speaker. aggregate_trials computes speaker-clustered FAR/FRR with Wilson CIs, auto-attribution coverage, and alignment attrition counts.

Attribution evaluation driver (tools/eval_attribution.py)

A maintainer tool for offline attribution scoring. Four subcommands:

• protocol: inspect/validate a protocol manifest.
• align: reads a self-contained input manifest pointing to gold RTTMs, hypothesis RTTMs, per-label match evidence, and an enrollment map. Runs align_slots + build_trials, writes a trials JSON with alignment provenance per meeting. Turn eligibility is sourced from the evidence (eligible_turns), not inferred from RTTM segment counts. Emits warnings to stderr for meetings without evidence or orphan evidence labels.
• score: trials JSON in, attribution metrics JSON out. Records the effective MatchingGates in the output for reproducibility. Optionally accepts a --gates override.
• report: renders one or more metrics JSONs into a dated Markdown report. Multiple runs compute a noise-floor spread. The report honestly states limitations (close-talk only, baseline not certification, Wilson CIs descriptive only).

All outputs use sort_keys, allow_nan=False, and schema_version: 1. The driver wraps all I/O and deserialization errors as EvalError (exit code 2). A frozen regression pack (tests/parity/fixtures/attribution/) commits synthetic trials and expected metrics so scorer determinism is CI-gated without a GPU.

uv run python tools/eval_attribution.py protocol --manifest protocol.json

uv run python tools/eval_attribution.py align \
    --manifest align-input.json --out trials.json

uv run python tools/eval_attribution.py score \
    --trials trials.json --out metrics.json

uv run python tools/eval_attribution.py report \
    --run metrics.json --date 2026-09-05 --out report.md

Calibrating the auto-display band (issue #114)

The three-band policy (voxint.speakers.policy) auto-displays a proposal only when it clears the grounded gate. Issue #114 calibrates that gate against corpus gold with a rule fixed before any score is seen. The rule text lives verbatim in the voxint.harness.calibration module docstring; the tooling below applies it.

Trials and clusters

build_trials tags every scoreable slot with a detail:

detail	gold speaker enrolled	machine top candidate	counts toward
genuine	yes	equals the gold speaker	FRR, coverage
impostor_closed	yes	a different roster speaker	FAR
impostor_open	no	any roster speaker	FAR
unscoreable	any	none, or alignment failed	attrition only

A slot with a gold speaker who is not on the roster used to be unscoreable; that hid open-set false accepts, the harm the auto-display band exists to bound.

Independence is a cluster count where a cluster is one gold speaker. The floor (MIN_INDEPENDENT_CLUSTERS = 50) applies to impostor clusters, because the certified claim is about false accepts. Genuine clusters are reported alongside with descriptive intervals and carry no floor: the AMI corpus has at most 35 recurring speakers, so a symmetric floor could never be met there. Both counts appear in check_independence, sweep points, and the attribution summary, and the FAR bound is the one-sided 95% Wilson upper bound at cluster level (wilson_upper_one_sided).

Protocol roles and the dev/confirm split (schema 2)

tools/generate_attribution_protocol.py --open-set N --gold-rttm-dir DIR writes a schema-2 manifest where every meeting carries a role (enrollment, test_genuine, test_open) and a split (enrollment, dev, confirm):

• Open-set meetings are AMI meetings with no cross-session speaker and a gold RTTM, picked by ascending SHA-256 of split_seed:meeting_id. Every participant is un-enrolled, so their slots become impostor_open trials.
• Dev and confirm are assigned per base session from metadata only. Test sessions that share a cross-session speaker form one component, so a genuine speaker never straddles the halves; components and open-set sessions are ordered by seeded hash and alternated so both halves are balanced.
• validate_split_honesty fails the generation on any leak (a speaker in both halves, an enrollment meeting in a scored split, an enrolled speaker inside an open-set meeting).

Schema-1 manifests are refused with a regenerate message. Align inputs and trials carry meeting_role and meeting_split; enrollment meetings are never scored.

One GPU pass, offline roster and matching

The corpus is run through the pipeline once with an empty roster and AUTO_ENROLL=false, so every run stores its per-turn embeddings and records no_roster evidence. Everything after that is CPU work through production code:

uv run python tools/build_attribution_roster.py \
    --protocol protocol.json --runs runs.json \
    --gold-rttm-dir gold_rttm --out-dir roster/

uv run python tools/rematch_runs.py --protocol protocol.json --runs runs.json

build_attribution_roster.py gold-aligns each enrollment run, keeps slots that pass the aligner's purity, coverage, margin, and eligibility floors, and enrolls the single best slot per cross-session speaker through the operator enrollment path (enroll_new_speaker, replay-safe idempotency keys). It writes enrolled_speaker_map.json and roster_fingerprint.json, and refuses to run on a roster that already holds foreign speakers unless --allow-existing. rematch_runs.py calls refresh_run_matches for the test-role runs only; enrollment-role runs are excluded by design. Because the matcher core is evaluate_run, the rematched evidence is exactly what the worker would have written with that roster.

Select on dev, certify on confirm

uv run python -m tools.calibrate_policy select \
    --trials attribution-trials.json --out selection.json

uv run python -m tools.calibrate_policy certify \
    --trials attribution-trials.json --candidate selection.json \
    --out certification.json

select scores the 32 pre-registered candidates (grounded cosine x grounded margin, everything else held at the base gates) on the dev trials. A candidate is feasible with zero auto_wrong and a cluster-level one-sided FAR upper bound at or below 5%; among feasible candidates the rule maximises genuine-cluster coverage, then minimises review load, then takes the stricter gate. No feasible candidate is a NO_DECISION.

certify scores the locked candidate once on the confirm trials. CERTIFIED requires at least 50 impostor clusters, zero auto_wrong, and the FAR bound; anything else is NO_DECISION with every failing reason (insufficient_impostor_clusters, auto_wrong_observed, far_bound_exceeded). The output also carries the PRE (base gates) and POST (candidate) tallies on the identical trials, the full band-change listing, FRR, coverage, and descriptive Wilson intervals, so the report is a gold PRE/POST diff rather than an opinion. select and certify filter on meeting_split; sweep and compare read the same file unfiltered for exploratory use.

AMI corpus calibration result

The first calibration run on 170 AMI Mix-Headset meetings returned NO_DECISION: the confirm split had 49 impostor clusters (floor: 50) and a one-sided 95% Wilson FAR upper bound of 5.23% (ceiling: 5.00%). Zero auto_wrong across all 311 scoreable trials. The 50-cluster floor was imported from the recurrence viability check and is internally inconsistent with the Wilson method: zero errors need at least 52 clusters to clear 5%. Grounded-gate defaults remain at their pre-calibration values (grounded_min_cosine=0.70, grounded_min_margin=0.08). Full analysis: docs/reports/attribution-calibration-2026-09-13.md. Evidence pack: tests/parity/fixtures/attribution/calibration/.

Feeding the harness from live runs (voxint.harness_export)

The harness scores files; it never reads a database. voxint.harness_export is the one package allowed to read the database and render live pipeline runs into these input shapes, so it lives OUTSIDE voxint.harness to keep that package DB-free. It exports STORED evidence only: it re-uses the matcher's own eligibility and centroid helpers over the stored per-turn vectors, and never re-runs TitaNet or the matcher decision, so an exported baseline reflects production numerics exactly.

• name_accuracy_items(session, run_ids, *, truth_anchoring) renders one name-accuracy item per run, one slot per diarization label. assigned_name is the matcher's automatic attribution, a grounded cosine proposal, read from the machine evidence INDEPENDENTLY of the read-time resolution: a label a human later adjudicated still reports what the machine would have shown, so a human ruling never masks the prediction being measured. truth is the human ruling (assign, __ABSTAIN__ for exclude, __NEITHER_DETERMINABLE__ for unknown, and absent for a label no human confirmed). Each slot also carries a match provenance block (the full accept/reject/ineligible decision, with margin null for a single-speaker roster, never Infinity) that the scorer ignores but a later confidence-policy pass can re-derive any band from. truth_anchoring records whether the truth was fixed independently of the proposal (a corpus) or after the operator saw it (their own material); the database cannot prove this, so the caller declares it.
• agreement_enrollment(session, embedding_space, ...) and agreement_slots(session, run_ids, ...) render the enroll-and-re-identify contract. Voiceprints are the active roster's production centroids; per-voice slot vectors are the production label centroids. held_out is asserted only when every contributing enrollment row records its source run, and source_item_ids lists those runs, so the harness leakage gate abstains on any item a voiceprint was built from.
• evidence_snapshot(session, settings, run_ids, *, exported_at, git_sha=None) records the code version, embedding-space identity, matching gates, and roster digest beside a baseline. It is an EXPORT-TIME snapshot, not a historical replay: the database does not retain the gates or roster centroids used when each run was matched, so the gates are labelled gates_at_export and the roster digest is an export-time fingerprint. A baseline whose snapshot no longer matches the live gates or roster is stale and must be regenerated.

The DB to JSONL mapping is unit-tested and round-tripped through the real score name-accuracy and score agreement commands.

Driving an export (tools/export_match_evidence.py)

The maintainer tool tools/export_match_evidence.py selects runs and writes the files. It reads a small run-selection manifest, calls the exporters above, and writes the artifacts atomically with the same deterministic, no-NaN serialization the harness uses, so a repeated export from an unchanged database is byte-for-byte identical. It is a maintainer tool, deliberately NOT a voxint subcommand: the score command stays DB-free by contract, so the one piece that reads the database sits outside it (alongside tools/qualify_local_llm.py).

The manifest (schema_version 1) declares one required embedding_space shared by every artifact and at least one lane:

{
  "schema_version": 1,
  "embedding_space": "titanet-large-v2",
  "name_accuracy": {
    "truth_anchoring": "independent",
    "run_ids": ["<run-uuid>", "..."]
  },
  "agreement": {
    "runs": [
      {"run_id": "<run-uuid>", "kind": "curated", "host_id": "<speaker-uuid>"},
      {"run_id": "<run-uuid>", "kind": "negative_control"}
    ],
    "roster_speaker_ids": ["<speaker-uuid>", "..."]
  }
}

truth_anchoring is independent (a corpus annotated without seeing Voxint's proposal) or post_proposal (the operator ruled after seeing it). A curated agreement run must name its host_id; a negative_control run must not. roster_speaker_ids is optional and defaults to the whole active roster. Run it with:

uv run python -m tools.export_match_evidence \
  --manifest baseline-selection.json --out-dir docs/reports/baseline-export

It writes snapshot.json always, name_accuracy_items.jsonl for the name-accuracy lane, and enrollment.json plus agreement_slots.jsonl for the agreement lane. It records the current git HEAD in the snapshot and refuses to run on a working tree with uncommitted tracked changes (pass --allow-dirty to override), so the snapshot's code sha means what it says. The agreement thresholds file that score agreement also needs is a calibration artifact, not DB-derived, so the driver does not produce it.

Relation to the private ancestors

The cores are fresh implementations of scorers developed in the private upstream project. Deliberate public-API divergences: identity ids are opaque strings (not integers), name normalization is Unicode NFKC + casefold (not ASCII-lowercase), goldset hashing is SHA-256 (not MD5), the surface reconciliation layer (private storage schema) was not ported, and gate-metrics key names are generic (baseline/candidate/audit).