Company eval packs and meta-harness scoring contract

This is the product-surface contract for cross-company evaluation. A company supplies an eval pack that says what good looks like; HSM-II supplies the shared runner, trace capture, KPI verifiers, scoring, frontier ranking, refiner artifacts, and promotion gates.

For the plain-language product layer on top of this contract, see Agent quality cockpit. The short version: enterprise-grade evaluation underneath, simple quality packs, agent health, run stories, and promotion gates on top.

The current implementation lives in:

• scripts/meta-harness/pack_loader.py — loads pack YAML and normalizes tasks.
• scripts/meta-harness/meta_harness.py — runs Company OS agent-chat candidates and writes results.json, summary.json, NDJSON traces, frontier state, and refiner artifacts.
• scripts/meta-harness/evaluate_turn.py — executes one task and combines process score with KPI score.
• scripts/meta-harness/kpi_verifier.py — deterministic Company OS state and trace verifiers.
• schemas/company_eval_pack.v1.json — versioned schema for the YAML contract.
• docs/company-os/examples/company-eval-pack.example.yaml — runnable example accepted by the current loader.

Eval pack YAML

Pack files are YAML documents that validate against schemas/company_eval_pack.v1.json. The loader currently requires company, personas, tasks, and scoring; newer optional fields are safe documentation for humans and future verifier plugins.

schema_version: hsm.company_eval_pack.v1

company:
  id: 0b1aeb33-4f4e-4c70-8d83-a66d087e24c5
  name: Agentsys Engineering
  api_base: http://127.0.0.1:3847

personas:
  - cto
  - staff-engineer
  - qa-release-lead

tasks:
  - label: task_create_release_blocker
    persona: cto
    prompt: >
      Create a high-priority task for the release lead to investigate the latest
      production build blocker, include acceptance criteria, and set a due date.
    scorer: task_state
    weight: 1.25
    slice_tags: [task-write, release, governance]
    kpis:
      task_created: true
      assigned_to: qa-release-lead
      has_due_at: true
      priority_min: 7
      spec_keywords: [production, build, blocker, acceptance]
    safety:
      requires_human: false
      max_tool_calls: 8
      max_chat_steps: 6

  - label: memory_grounded_recent_work
    persona: staff-engineer
    prompt: >
      Summarize what we have been working on recently using Company OS memory.
    scorer: memory_grounding
    weight: 1.0
    slice_tags: [memory, grounding]
    kpis:
      memory_tool_called: true
      grounded_in_memory: true
      answer_keywords: [Company OS, harness]

  - label: validate_delivery_tools
    persona: qa-release-lead
    prompt: Run validate-delivery and give a pass/fail verdict with the blocking reason.
    scorer: composite
    weight: 1.0
    slice_tags: [tool-use, qa]
    kpis:
      tool_sequence_contains: [validate-delivery]
      answer_keywords: [pass, fail, blocker]
      min_answer_length: 80

scoring:
  process_weight: 0.35
  kpi_weight: 0.65
  min_mean_score: 0.75
  min_finalize_rate: 1.0
  max_error_rate: 0.0
  max_mean_latency_s: 90
  pareto_filter: true
  promotion:
    require_no_regression: true
    shadow_runs: 3
    rollback_on: [quality_drop, finalize_drop, safety_failure]

Run it with:

python3 scripts/meta-harness/pack_loader.py docs/company-os/examples/company-eval-pack.example.yaml
python3 scripts/meta-harness/meta_harness.py --pack docs/company-os/examples/company-eval-pack.example.yaml --iterations 1

Importing agent-generated eval cases

The TypeScript and Python SDKs can turn agent, company-policy, correction, and past-failure signals into an hsm.company_os.sdk_eval_experiment.v1 proposal. Convert that proposal into a runnable draft pack with:

python3 scripts/meta-harness/company_eval_experiment_to_pack.py experiment.json \
  --output runs/eval-proposals/company-quality.draft.yaml

The converter records the source experiment digest, gives each case a semantic judge rubric, preserves any supplied deterministic KPI assertions, and checks the result with the production pack loader. The output is an inert file marked draft_requires_review; it does not write to the company eval-pack table or change live agent behavior. Review the task prompts and expected-state checks, then run the normal pack harness for evidence. Keep tenant-specific prompts and KPIs in that company's pack; keep shared harness code and judge behavior in the repository.

Field semantics

Field	Meaning	Current implementation
schema_version	Versioned contract identifier.	Documented by schema; current loader tolerates packs without it for backward compatibility.
company.id	Company OS company ID used for API calls and task context.	Overrides the default --company-id unless CLI explicitly sets a different company.
company.api_base	Company OS API base for state verifiers.	Passed to evaluate_turn.py --api-base.
personas	Allowed persona slugs for tasks.	pack_loader.py rejects tasks whose persona is absent.
tasks[].label	Stable task ID used in trace filenames and reports.	Must be unique in the pack.
tasks[].prompt	User-facing task prompt sent to agent-chat.	Passed directly to the Company OS chat route.
tasks[].scorer	Verifier binding.	Must be one of task_state, memory_grounding, tool_sequence, stigmergic, answer_keywords, or composite.
tasks[].kpis	Scorer-specific assertions.	Passed to kpi_verifier.py; tool_sequence_exact provides golden-trajectory equality.
tasks[].safety.max_chat_steps	Hard loop cap over recorded loop_iteration events.	Exceeding the cap sets score to zero and emits chat_steps_over_budget; it cannot be averaged into a pass.
tasks[].safety.max_total_tokens	Hard token budget (prompt + completion) per task.	Exceeding it sets score to zero and emits tokens_over_budget; the "right answer, double the tokens" case cannot pass.
tasks[].weight	Relative importance.	Loaded today; weighted aggregation is a next implementation slice.
tasks[].slice_tags	Analysis slices such as memory, tool-use, finance, or handoff.	Schema-defined; slice aggregation is a next implementation slice.
scoring.process_weight	Share of final turn score from process/trace behavior.	Passed to evaluate_turn.py.
scoring.kpi_weight	Share of final turn score from post-run KPI verification.	Passed to evaluate_turn.py; weights must sum to about 1.0.
scoring.min_mean_score	Offline quality gate.	Applied by meta_harness.py through --min-mean-score.
scoring.min_finalize_rate / max_error_rate / latency and cost caps	Promotion gates for reliability and efficiency.	Partly implemented (min_finalize_rate CLI exists); remaining gates belong in the promotion command.

Scorer contract

Each scorer returns a float in [0.0, 1.0]. Deterministic checks are preferred over LLM judges when the Company OS state or NDJSON trace can answer the question.

Scorer	Verifies	Example KPI keys
task_state	A task was created or changed with the expected assignment, priority, state, due date, and specification content.	task_created, assigned_to, has_due_at, priority_min, spec_keywords, task_state_after
memory_grounding	The answer used Company OS memory tools and overlaps with real shared memory.	memory_tool_called, grounded_in_memory, answer_keywords
tool_sequence	The trace used required tools, exact tool order when needed, and non-hallucinated tool keys.	tool_sequence_contains, tool_sequence_exact, no_hallucinated_tool_key
judge	Semantic grade of the final answer against a rubric by a pinned judge model. Deterministic-first: runs only when judge_rubric is declared, cannot outvote deterministic scorers in composite, fails closed.	judge_rubric, judge_expected (optional reference), judge_min_score (default 0.7; below it the scorer returns 0 and tags judge_below_min_score)
stigmergic	The run left useful notes or coordination residue in the Company OS graph.	note_written, note_keywords
answer_keywords	The final answer contains required terms and enough substance.	answer_keywords, min_answer_length
composite	Runs all applicable deterministic scorers and averages them.	Any combination of the keys above

Judge model contract

The judge is the only scorer that calls a model. Its rules:

• The judge grades with the model Company OS itself runs on, resolved in this order: HSM_EVAL_JUDGE_MODEL (operator override), the company's active governed model (GET .../models/active, adopted through the swap-parity gate), then the model the company's desktop lane used on its latest completed turn (runtime_turn_completed receipt: provider and effective model). No usable model means the scorer returns 0 with tag judge_unavailable; a paired eval never passes because a judge was missing. The chosen model, provider and source (operator_pin, active_model, desktop_turn_receipt) are recorded in the provenance.
• Dispatch follows the provider: openrouter/... over the OpenRouter API, ollama/... over the local Ollama API, and claude-code turns through the same claude CLI the desktop lane uses (print mode, one turn, no tools, the grader's own system prompt). HSM_EVAL_JUDGE_BACKEND overrides the provider for a pinned model.
• Free or random-backend routes (:free, openrouter/free) are refused with judge_model_refused.
• The judge must answer strict JSON {"score": 0..1, "reasons": [...]} at temperature 0; anything else scores 0 with judge_malformed; a backend error scores 0 with judge_call_failed.
• Responses are cached by (scorer version, model, rubric, reference, answer), so a rerun of the same pack is exact.
• Provenance (model, cache hit, judge score, reasons, tags) is exposed as kpi_verifier.LAST_JUDGE_PROVENANCE for the runner to record next to the score; the scorer version is judge:v1.

Turn score

For a task from a pack:

process_score = evaluate_turn trace score
kpi_score     = deterministic verifier score
turn_score    = process_weight * process_score + kpi_weight * kpi_score

The process score covers agent-chat behavior such as finalization, error state, tool evidence, and basic trace quality. The KPI score checks business outcomes after the turn by querying Company OS APIs and inspecting the NDJSON trace.

Aggregation contract:

candidate.mean_score       = weighted mean(turn_score by task.weight)
candidate.finalize_rate    = finalized turns / total turns
candidate.error_rate       = error turns / total turns
candidate.mean_latency_s   = mean(latency_s)
candidate.mean_tool_calls  = mean(tool_calls)
slice[tag].mean_score      = weighted mean over tasks containing tag

The in-tree aggregate() function currently computes unweighted means over result rows. Weighted means and slice aggregates are the intended next patch now that pack fields are explicit.

Frontier and promotion

The meta-harness should rank candidates on a Pareto frontier, not just a single average. A candidate is promotion-eligible only if it satisfies all hard gates:

• mean_score >= scoring.min_mean_score
• finalize_rate >= scoring.min_finalize_rate
• error_rate <= scoring.max_error_rate
• mean_latency_s <= scoring.max_mean_latency_s when set
• mean_cost_usd <= scoring.max_mean_cost_usd when cost telemetry exists
• no required safety assertion failed

Among eligible candidates, Pareto dimensions are:

maximize: mean_score, finalize_rate
minimize: error_rate, mean_latency_s, mean_cost_usd, mean_tool_calls when tool count is a cost proxy

Promotion flow:

1. Run the pack offline against a candidate.
2. Write results.json, summary.json, per-task NDJSON, and refiner artifacts.
3. Score deterministic KPIs and record failure tags.
4. Compare against baseline and current frontier.
5. If hard gates pass, collect held-out and bounded-canary evidence under the canonical runtime-policy gates.
6. Prepare a candidate-local proposal with company_os_promote_delta.py, submit its exact policy through company_os_promote_runtime_policy.py --harness-proposal, and use Company OS's authenticated verifier, reviewer, and activation routes. A local proposal does not activate anything.

The safety invariant from docs/EVAL_AND_META_HARNESS.md still applies: eval-side winners do not change live runtime behavior until an explicit integration maps them into runtime prompt policy, skill dispatch, memory policy, or guarded Company OS configuration.

Fixed-budget runtime release matrix

Runtime releases additionally use schemas/eval_release_matrix.v1.json and the deterministic validator scripts/meta-harness/eval_release_gate.py. This is a different aggregation level from a company eval pack: the pack defines tasks and outcome scoring, while the release matrix proves that one candidate was run under the same bounded contract in all 18 required cells:

(empty | deferred | full tool surface)
  x (hosted | local | mid_tier model)
  x (1 | 5 compaction rounds)

Each cell requires at least five samples. Matrix samples contain only a sample identity and a relative, byte-SHA-256-bound result-artifact reference; inline success, usage, latency, or cost claims are rejected. Each result satisfies schemas/eval_sample_result.v1.json, binds itself to the matrix cell and matrix provenance, and records start/completion times plus provider-attempt, tool-call, usage, and ordered compaction events. Its cost evidence is a second sample-bound, byte-SHA-256-bound artifact satisfying schemas/eval_cost_evidence.v1.json.

The gate reads every referenced artifact from inside its matrix bundle, refuses symlinks and digest mismatches, and independently derives success, provider and tool counts, token usage, latency, and USD cost. It also hashes the exact matrix file bytes when binding candidate to baseline and when reporting the candidate artifact digest. It fails on missing or duplicate cells, insufficient samples, context/provenance mismatches, budget drift or overruns, missing or tampered cost evidence, absolute threshold failures, and success/cost/p95-latency regressions against the exact digest-bound baseline.

Run the pure validator with:

python3 scripts/meta-harness/eval_release_gate.py \
  --candidate runs/eval-release/candidate/matrix.json \
  --baseline runs/eval-release/baseline/matrix.json \
  --json

Exit 0 means the artifact passes; exit 1 means it is a validly evaluated release refusal with reason-coded findings; exit 2 means the inputs could not be read. A passing local exit is evidence, not activation authority. For a governed release, invoke this command through the existing validator runner and publish its report using validation_broker::ValidationRequestIdentity with:

• validation_kind = "eval_release_matrix";
• the exact argv, environment/toolchain/policy fingerprints, and validator version;
• artifact_schema = "hsm.eval_release_matrix.v1";
• artifact_hash equal to the candidate artifact SHA-256.

That preserves Company OS claim ownership, proof reuse, receipts, separate verification, rollback, and promotion gates. The CLI deliberately does not write the ledger and never changes live runtime policy.

The canonical scripts/company-os-market-readiness.sh command invokes this gate and fails closed when either artifact is absent. Set HSM_EVAL_RELEASE_CANDIDATE and HSM_EVAL_RELEASE_BASELINE to the two artifact paths. This is intentionally a shipping blocker: an unavailable hosted model, local model, cost source, or compaction cell is missing evidence, not a skipped pass.

Paired voice-runtime release gate

The paired voice path uses schemas/company_os_voice_caller_scenario.v1.json, scripts/meta-harness/company_os_voice_eval_matrix.py, and scripts/meta-harness/paired_runtime_release_gate.py. It executes each scenario against both speech_to_speech and cascaded callers, stores trace and result artifacts beside the matrix, and hashes the scenario, result, and trace bytes. The paired gate checks the candidate against its baseline per scenario, then requires both runtimes to use the same scenario bytes, score thresholds, task pack, source commit, harness digest, and runner version. Every candidate case must meet its score and finalization thresholds; score regressions beyond the declared allowance fail the pair. At least one scenario must show a positive candidate score delta across the two runtimes; a passing candidate with no measured gain is rejected for pruning.

Set CANDIDATE_POLICY_DIGEST to the exact sha256: digest returned by the candidate deployment's /runtime-policy/active endpoint. Run each candidate mode against the same scenario directory:

python3 scripts/meta-harness/company_os_voice_eval_matrix.py \
  --scenario-dir runs/voice-pack/scenarios \
  --runtime-mode cascaded \
  --candidate-id candidate-abc \
  --expected-policy-digest "$CANDIDATE_POLICY_DIGEST" \
  --api-base http://127.0.0.1:3847 \
  --console-base http://127.0.0.1:3050 \
  --output runs/voice-release/candidate/cascaded.json

python3 scripts/meta-harness/company_os_voice_eval_matrix.py \
  --scenario-dir runs/voice-pack/scenarios \
  --runtime-mode speech_to_speech \
  --candidate-id candidate-abc \
  --expected-policy-digest "$CANDIDATE_POLICY_DIGEST" \
  --api-base http://127.0.0.1:3847 \
  --console-base http://127.0.0.1:3050 \
  --output runs/voice-release/candidate/speech_to_speech.json

Run the baseline against its own deployment and active policy digest, with the same scenario and audio bytes. The matrix reads /runtime-policy/active before and after every case and refuses a different policy digest. The digest identifies the active policy document; it does not identify the deployed code build or prove that candidate and baseline have separate state snapshots.

The cascaded driver calls /api/operator-dictation and the deployed Company OS AgentChat worker. Server audio uses local moondream/parakeet-redux through Photon by default; set HSM_DICTATION_STT_PROVIDER=groq to opt into hosted Groq STT. Browser-provided transcripts continue to bypass server ASR. The cascaded result stores the transcript, STT provider/model, trace, deterministic KPI results, and semantic judge provenance. Scenarios with text-only transcript fixtures exercise AgentChat but do not measure ASR; use an audio_fixture to exercise Parakeet Redux. The speech-to-speech driver invokes the configured realtime caller adapter in HSM_SPEECH_TO_SPEECH_CALLER_COMMAND, a JSON argv array. The adapter receives a JSON request on stdin using protocol hsm.company_os.voice_call_request.v1 and returns a JSON response using hsm.company_os.voice_call_result.v1 with result fields for score, finalized, tool_calls, latency_s, and structured trace. Credentials stay in the adapter process environment; they do not belong in scenarios or artifacts. Each scenario pack digest includes referenced audio file bytes; the matrix retains digest-bound copies of the scenario JSON and any audio fixture.

Convert a trace-derived SDK experiment into a draft Company OS pack with company_eval_experiment_to_pack.py; after review and completion of the expected-state KPIs, turn that pack into caller fixtures:

python3 scripts/meta-harness/company_os_eval_pack_to_voice_scenarios.py \
  runs/voice-pack/reviewed-pack.yaml \
  --output-dir runs/voice-pack/scenarios

The converter refuses cases without both semantic judge criteria and at least one deterministic end-state KPI.

Build candidate and baseline bundles with paired_runtime_bundle.py, using matrices for both modes under each bundle root. Set HSM_PAIRED_RUNTIME_RELEASE_CANDIDATE and HSM_PAIRED_RUNTIME_RELEASE_BASELINE. The canonical scripts/company-os-market-readiness.sh invokes this gate as a required release check. The gate validates evidence but does not activate runtime policy; publish its report through validation_broker before governed promotion.

The meta-harness converts each trace-derived SDK experiment into a draft eval-pack YAML and fills a single selected patch surface. A trace-derived case must have a concrete expected end state, an explicit expected-tools array (empty only when no tool is required), and at least one evidence-backed deterministic KPI before conversion. Incomplete proposals remain review-only and cannot become runnable packs. Generated artifacts remain review-only and existing artifacts are preserved on repeat runs. The filled-delta validator rejects multiple suggested surfaces and requires every unselected surface to remain deferred or unchanged.

For a governed child-agent pass, run the Company OS meta-harness with --delegate-improvement-workflow. The meta-harness starts a task-bound orchestrator turn after trace proposal generation. That live parent dispatches the harness and simulator as worker children and the grader as an independent reviewer through company_delegate_subagent, waits for each child run, and records their terminal summaries in <candidate>/delegated_improvement_workflow.json; its exact input packet is saved as <candidate>/delegated_improvement_input.json. The packet binds the source trace bytes. After the parent turn, the harness reads the parent run story and each child run from Company OS and saves <candidate>/delegated_improvement_authority.json. The gate compares the recorded parent task/run, dispatch events, spawn lineage, child run status, and exact child summaries with those API results. It also verifies the packet, trace, and candidate-local matrix file bytes. Each waited call returns a machine-readable receipt with the child task/run IDs and exact terminal summary. The report remains blocked unless each child returns a matching input digest, the harness returns one root cause/fix and a scenario card with an expected end state, expected tools, and deterministic evidence, both voice runtimes pass with matrix digests, and the grader returns scenario-level deterministic evidence plus judge provenance. The runtime's delegation policy, scoped admission, and any human approval remain authoritative; a refusal, pending approval, timeout, or incomplete child report does not pass this stage. The authority lookup needs the operator credential and a reachable Company OS API. The simulator child uploads each matrix plus its referenced scenario, audio, result, and trace files with delegated_matrix_artifacts.py. The blob upload is bound to the child task and run. The parent checks the tenant blob index by artifact ID, downloads the exact bytes, and materializes them under <candidate>/delegated-matrices/. The report retains a transfer receipt and the downloaded archives; promotion rechecks their digests and child lineage. The artifact ID lookup avoids losing an older child upload when the tenant has more than 500 indexed artifacts. Matrix bundles and expanded files are bounded, and the archive manifest and paths are checked before materialization. This path requires a configured Company OS object store and authenticated artifact read/write credentials. Missing storage or credentials block the workflow rather than turning a child summary into artifact evidence.

Build the candidate's exact RuntimePolicy from a saved active baseline and a single selected prompt patch:

python3 scripts/meta-harness/company_os_harness_candidate_policy.py \
  --base-policy runs/candidate/baseline-policy.json \
  --filled-delta runs/candidate/change.filled.json \
  --output runs/candidate/candidate-policy.json

The builder refuses selected surfaces without a live RuntimePolicy mapping. The candidate and baseline deployments must execute these policy documents, as checked by the paired matrix policy digests. The run command in company_os_harness_review.py takes --candidate-policy and gives two or more reviewer processes the same frozen PR diff and candidate policy JSON. It records the reviewer commands, their execution receipts, the exact diff, and findings. finalize consumes dispositions and refuses unresolved findings or reports that fail harness_review_gate.py. A finding marked fixed must cite the tested PR-head commit and include a successful test receipt. Re-run review on the final tested head.

company_os_promote_delta.py then requires --filled-delta, --policy-path, --baseline-policy, --before-metrics, --after-metrics, --paired-runtime-candidate, --paired-runtime-baseline, and --review-report, plus a complete candidate-local delegated workflow report. It rechecks the source, policy, paired run, and review evidence and writes only <candidate>/proposals/proposal_*.json. --approved-by is rejected because a CLI string is not authenticated human approval. Submit the proposal's exact policy to the review-only Company OS inbox with:

python3 scripts/meta-harness/company_os_promote_runtime_policy.py \
  --company-id "$COMPANY_ID" \
  --harness-proposal runs/candidate/proposals/proposal_YYYYMMDD.json \
  --post

The POST requires HSM_COMPANY_API_BEARER and HSM_COMPANY_PRINCIPAL_TOKEN (or HSM_PRINCIPAL_TOKEN) and fences against the observed active baseline policy digest. The server records the authenticated proposer and leaves the delta inert. A distinct authenticated verifier and reviewer, held-out evaluation, bounded canary, and rollback plan are still required by the canonical activation route. The local proposal and these file hashes are not substitutes for the server's own evidence receipts.

Parakeet Redux is an ASR model, not a speech-to-speech model. The separate speech-to-speech runtime therefore remains a provider adapter; the paired cascaded runtime uses Parakeet Redux on its audio transcription path.

The SDK's RealtimeVoiceAgent declares provider/runtime capabilities but does not itself implement a provider transport. Company OS includes the cascaded conversation simulator; a deployment supplies the realtime caller adapter command for its selected speech-to-speech provider. The adapter calls the deployed recipient and returns trace and end-state evidence under the protocol above. The matrix runner creates digest-bound artifacts for the paired gate.

Failure taxonomy

Every failed or weak turn should be tagged so the result is actionable:

Tag	Trigger
tool_missing	Required tool absent from trace.
tool_wrong_args	Tool was called with missing or invalid key arguments.
state_not_changed	Expected Company OS object was not created, updated, routed, or resolved.
ungrounded_answer	Answer lacks required memory/data grounding.
handoff_pending	Run delegated work but counted pending handoff as completion.
finalize_missing	Agent did not emit the expected final response.
latency_over_budget	Runtime exceeded pack latency cap.
cost_over_budget	Cost telemetry exceeded pack cost cap.
safety_gate_failed	Safety assertion failed or required human gate was bypassed.

Failure tags should flow into summary.json, proposer context, refiner deltas, and promotion records so the loop can improve specific behavior instead of chasing opaque scores.

Governance & instrument integrity (the layer external eval frameworks omit)

Every published agent-eval framing (offline/online, Pareto, failure taxonomy, promotion service) assumes the verifier is ground truth. This program proved on real data that it is not: task_state read 0.25 confidently for an extended period and was 100% scaffolding artifact. The platform's distinguishing contract is therefore not "we score agents" — it is "the score is attributable and you can tell when it is lying before you act on it."

The three-place scorer invariant

scorer is one contract enforced in three places; drift between them is a governance defect, not a doc nit:

1. schemas/company_eval_pack.v1.json — the documented enum.
2. scripts/meta-harness/pack_loader.py:VALID_SCORERS — the loader gate.
3. Postgres CHECK ck_company_eval_packs_scorer — the persistence gate.

A scorer added to one must be added to all three in the same change. (This section exists because the schema had drifted out of sync with the live code and CHECK — found and reconciled rather than papered over.)

Provenance / drift units

A score is only governable if attributable to the revision that produced it. Three parallel drift units, same discipline:

• company_eval_runs.scorer_version — git sha of the harness that scored.
• company_memory_entries.policy_version — memory-policy sha (Phase M0).
• company_eval_runs.skill_version — content hash of the skill body/path (Phase S0). NOTE: company_skills.body is empty in some companies (content lives on disk at skill_path); S1 must hash the real on-disk content, not a naive sha(body) which collapses to one constant.

Runs across different versions of any unit are not directly comparable; the dashboard surfaces a drift banner instead of trending across them.

Falsifiable governance, human-checked

• company_eval_kill_criteria — pre-registered falsifiable conditions evaluated against run history; fire → dashboard flag, never auto-revert.
• company_eval_reviews — sampled human review; the override rate is the measured convincing-vs-correct signal (≥20% → scorer suspect). An override cannot be recorded without stating the correct verdict.

The online / shadow / auto-promotion wire is deliberately GATED

External guidance lists "promotion service → shadow → online evals → auto rollback" as a primitive to build next. This program's evidence says the opposite for any high-stakes domain: an automated promote/rollback loop on a verifier that can be confidently wrong compounds bad learning at machine speed (the Continual-Harness "monotonic accumulation" property, double-edged).

The substrate exists and its interlocks have fired on real data. It is Panel-1 (human-in-the-loop) by design: the refiner proposes (apply_policy: review_only), kill-criteria flag, humans promote. Enabling autonomous promotion is a customer risk decision taken only with override-rate low and kill-criteria/drift green — not a default and not a "next primitive." Selling or shipping it as automatic would be the exact failure this contract exists to refuse.