Eval, meta-harness, and outer loop

This document is the canonical guide for when to use each tool, where artifacts land, and the contract between promoted harness JSON and the rest of HSM-II.

For Company OS agent-chat regression (locked NEXT_URL / HSM_URL, meta-harness vs LongMemEval, optional services/agent-chat-harness), see company-os/AGENT_CHAT_REFERENCE_HARNESS_PROFILE.md.

For cross-company declarative eval packs, YAML schema, KPI scorer bindings, and the Company OS agent-chat scoring/promotion contract, see company-os/eval-pack-and-scoring-contract.md.

For the paired, successive-round comparison of decisions with no learning versus only qualified Company OS learning, including outcome quality, regret, overrides, cost, evidence accuracy, and falsification control, see company-os/DECISION_LEARNING_BENCHMARK.md.

For standardized academic model-quality benchmarks through EleutherAI lm-evaluation-harness, see company-os/lm-evaluation-harness-lane.md.

For the optional production Test-Time Self-Reflection caller, trainer allowlist, sample-disjoint evidence, and crash-durable cleanup contract, see company-os/TTSR_GOVERNED_RUNTIME.md.

FlashEvolve (async evolution orchestration) — arXiv:2605.08520 targets wall-clock cost of GEPA/Meta-Harness loops via queued overlapping stages. HSM-II runs those stages synchronously today; FlashEvolve is the planned throughput layer. See company-os/ax-harness-evolution-stack.md.

For native SMB-style benchmarks (different harness), see HSM_NATIVE_BENCH.md.

For the opt-in Mercor APEX-Agents 1.1 external-runner lane, see eval/APEX_AGENTS_COMPANY_OS_LAB.md.

---

Which tool when

Tool	Role	Use it when
hsm-eval	Inner eval runtime — runs the benchmark suite with HsmRunner vs a vanilla LLM baseline, writes metrics and optional JSONL traces.	You want a single measurement pass, a fixed HsmRunnerConfig, or to validate a JSON harness file (e.g. after meta-harness) with --hsm-config.
hsm_meta_harness	Outer search (phase 1) — samples or loads many HsmRunnerConfig candidates, compares each to baseline, ranks by objective, exports Pareto frontier, optionally writes best_config.json and can promote a file to config/hsm_harness.default.json.	You want to search harness knobs (memory injection, budgets, thresholds, etc.) against the same verifiable tasks, not tweak weights.
hsm_outer_loop	Outer-loop infrastructure — compile gate, SQLite over runs/runs_index.jsonl, queries, proposer context for agents, external Rust benchmark batches (e.g. YC-bench, side repos).	You need to index/query past runs, feed a coding agent context from history, or run non-in-tree harnesses from JSON specs.
scripts/lm-eval-company-os.sh	Academic model-quality lane — wraps EleutherAI lm_eval presets and writes artifacts under runs/lm-eval/.	You want standardized MMLU/GSM8K/HellaSwag/TruthfulQA-style scores for model choice or release reporting.
Overeasy remote-sandbox experiment	Review-only filesystem durability lane — an opt-in 20-case Modal/FUSE probe under scripts/overeasy_experiment/ for recovery, branching, replay, receipts, credentials, and publication boundaries.	You want evidence for a possible remote workspace-backing adapter without changing the production runtime or VCS broker. Simulation is contract testing only and cannot pass the live gate.
Governed runtime-mechanism research	Outer search over typed runtime policy — seals search/held-out tasks and frozen economic/quality metrics, proposes versioned RuntimePolicyDelta candidates, and binds eval, release-matrix, lineage, failure-tag, and rollback evidence.	You want to compare runtime mechanisms such as observation projection without source mutation or automatic activation. See company-os/GOVERNED_RUNTIME_MECHANISM_RESEARCH.md.

Mental model: hsm-eval = one HSM-native experiment; hsm_meta_harness = many candidates + leaderboard; hsm_outer_loop = archive/DB/tooling around runs; lm_eval = standardized academic model quality, not Company OS workflow quality.

---

Contract: promoted config vs production runtime

• HsmRunnerConfig is the tunable policy object used by HsmRunner inside hsm-eval, hsm_meta_harness, and related eval binaries (hsm_native_eval, hsm_longmemeval, etc.).
• personal_agent, hsm_console, and the main Telegram/API agent stack do not automatically load config/hsm_harness.default.json or HSM_META_HARNESS_CONFIG. Their memory and tool behavior use different configuration paths.
• Therefore: meta-harness and hsm-eval are eval-side tooling. POST .../runtime-policy/import-best-config maps only fields with a real live consumer and records a source digest plus explicit unmapped/eval-only keys in a review-only RuntimePolicyDelta. It does not activate the artifact. A separate reviewer must approve it, a separate verifier must verify it, and only then may the delta activation route create an active runtime_policy_versions row consumed by agent-chat and workers through GET .../runtime-policy/active. A promoted best_config.json (or copied config/hsm_harness.default.json) is never itself a live runtime switch.

Practical implication: Treat best_config.json as the artifact you re-test with hsm-eval --hsm-config path/to/best_config.json and, if you change product behavior manually, as documentation of what worked on benchmarks — not as a switch that flips personal_agent today.

---

Validation claims: serializing expensive evals and reusing exact proof

Company OS pack evaluation can be gated by the validation broker (src/company_os/validation_broker.rs) via eval_harness::run_eval_with_claim(pool, company_id, pack_id, failure_class, &EvalClaimContext):

• Serialization — concurrent equivalent evaluations coordinate on the (company_id, "eval:<pack_id>:<failure_class>") key; exactly one claimant runs, the rest observe Busy.
• Proof reuse — a passed run publishes append-only proof carrying the artifact identity eval_result.json:sha256:<digest>. A later request whose canonical identity fingerprint matches exactly returns Reused(proof) without re-running.
• Drift refusal — the fingerprint covers Git head, workspace/dependency/ toolchain/policy fingerprints, feature flags, and environment. Any drift (e.g. a new git_head) forces a fresh run; stale proof is never returned.
• Fail-closed — failing or erroring evaluations release the claim without publishing proof, so nothing can "reuse" a failure as evidence.

Agents reach the same coordination through the governed HTTP surface (POST /api/company/companies/{id}/validation/claims[…/heartbeat|complete|fail]) or the company_validation_* tools; both paths enforce principal tokens and the company_os:validation delegation scope.

---

Where artifacts go

Default layout uses a runs/ directory at the repo root (override with --out-dir / --artifacts where supported).

hsm-eval

• With --artifacts <dir>: writes manifest.json, comparison outputs, and per-suite dirs with turns_hsm.jsonl, turns_baseline.jsonl, optional hsm_trace.jsonl, paths recorded under artifact_paths in the manifest.
• May append one line to runs/runs_index.jsonl (unless disabled) for outer-loop ingestion.
• Compaction-guideline collector: --hsm-compaction-pairs-out <path> runs matched full-context and compacted passes and writes the serialized divergence evidence. Add --hsm-compaction-company-id <uuid> to ingest that emitted artifact into the tenant's governed self-improvement proposal ledger; this requires HSM_COMPANY_OS_DATABASE_URL or DATABASE_URL. With multiple suites, the suite name is appended to each output filename.

Applied compaction rules cross from Company OS/Postgres into the database-free LCM host through a verified snapshot. Configure both HSM_LCM_GUIDELINE_SNAPSHOT_PATH and HSM_LCM_GUIDELINE_COMPANY_ID; setting only one, loading another company's snapshot, a malformed or oversized file, a symlink target, non-canonical rules, or a digest mismatch makes live LCM construction fail closed. Governed apply and rollback refresh the configured company's snapshot atomically.

hsm_meta_harness

• Default run directory: runs/run_<unix_timestamp>/ (or --out-dir).
• Per run:
  • baseline_by_suite.json (and baseline_metrics.json if a single suite).
  • Under cand_*/: candidate_result.json, per_suite.json, per-suite subdirs with hsm_metrics.json, comparison_report.json, turns_hsm.jsonl, turns_baseline.jsonl, optional hsm_trace.jsonl.
  • leaderboard.json, pareto_frontier.json, manifest.json.
  • If the confidence gate passes: best_config.json (full HsmRunnerConfig JSON).
• promote subcommand copies a harness JSON to config/hsm_harness.default.json by default (see hsm_meta_harness promote --help).

hsm_outer_loop

• external-batch / external: write results under paths given in the spec (e.g. runs/external_batch_<timestamp>.json).
• index-db: builds runs/runs.sqlite from runs/runs_index.jsonl (paths configurable).
• propose: emits ProposerContext JSON (e.g. for agent workflows).

Environment variables (see .env.example): HSM_RUNS_SQLITE, HSM_PARENT_RUN_ID, HSM_META_HARNESS_CONFIG, eval thresholds, etc.

---

Smoke recipe (copy-paste)

Prerequisites: Rust toolchain, repo clone, and any one LLM path the rest of the project uses (e.g. Ollama running locally, or OPENAI_API_KEY / ANTHROPIC_API_KEY set — same as for hsm-eval).

Minimal Ollama example:

cd /path/to/HSM-II
# Ensure your model is available, e.g.:
# ollama pull llama3.2
export OLLAMA_MODEL=llama3.2

1) Single eval (hsm-eval) — tiny slice

cargo run --bin hsm-eval -- --suite memory --limit 2 --verbose

2) Eval with artifacts (for Trace2Skill / inspection)

mkdir -p runs/smoke_eval
cargo run --bin hsm-eval -- --suite memory --limit 2 --artifacts runs/smoke_eval

3) Meta-harness — smoke search (small sample)

Meta-harness enforces a minimum task count unless you opt out:

cargo run --bin hsm_meta_harness -- \
  --candidates 2 \
  --bootstrap-runs 1 \
  --suite memory \
  --limit 2 \
  --allow-small-sample \
  --require-positive-ci=false

4) Outer loop — list runs

After a run that appended runs/runs_index.jsonl:

cargo run --bin hsm_outer_loop -- list-runs --index runs/runs_index.jsonl --limit 10

5) Validate a promoted or best_config.json with hsm-eval

cargo run --bin hsm-eval -- \
  --suite memory \
  --limit 2 \
  --hsm-config runs/run_<timestamp>/best_config.json

(Use the actual path to your best_config.json.)

---

Company OS agent-chat (Python scripts/meta-harness)

This is a separate harness from the Rust hsm_meta_harness binary: it drives the live Next.js Company Console routes POST /api/agent-chat-reply/stream and POST /api/agent-chat-reply, talking to hsm_console for tasks and worker execution (evaluate_turn.py defaults: Next http://127.0.0.1:3050, HSM http://127.0.0.1:3847).

Prerequisites

• cargo run -p hyper-stigmergy --bin hsm_console (or your usual Company OS API) reachable on 3847 (or override in evaluate_turn.py).
• cd web/company-console && npm run dev (or production build) on 3050 so the stream route exists.
• LLM keys the console expects (e.g. OpenRouter / OPENROUTER_API_KEY per web/company-console docs).
• Harness data directory: by default the script uses ~/.hsm/meta-harness/ when your home directory is writable. If not (e.g. Cursor sandbox, read-only CI), it automatically falls back to <repo>/.meta-harness/ (gitignored). Override anytime with HSM_META_HARNESS_DATA_DIR.

Timeouts

• Server-side worker telemetry waits are controlled by HSM_OPERATOR_CHAT_TELEMETRY_WAIT_EXEC_MS / HSM_OPERATOR_CHAT_TELEMETRY_WAIT_ANALYSIS_MS (see .env.example). JSON and NDJSON paths share the same caps; build-heavy skills get a longer minimum wait.
• Next route maxDuration and the browser fallback poll budget are defined in web/company-console/app/lib/operator-chat-timeouts.ts so hosted and UI behavior stay above those waits.

Commands

cd /path/to/hyper-stigmergic-morphogenesisII
# Quick smoke (default: `--paper-smoke` — three tasks aligned to Meta-Harness paper harness dimensions: tools + synthesis + memory recall; legacy: `MH_SMOKE_LEGACY=1` uses first N tasks)
./scripts/company-os-agent-chat-meta-harness-smoke.sh

# Same paper-aligned trio, invoked directly
python3 scripts/meta-harness/meta_harness.py --paper-smoke

# First three rows of `EVAL_TASKS` only (all engineering prompts)
python3 scripts/meta-harness/meta_harness.py --tasks 3

Belief state (experimental) — evaluate_turn.py also prints a belief_state object: a Beta-style summary over a single “task success” latent plus a VoI proxy for logging, aligned with Papamarkou et al. (Bayes-consistent decisions; HAL hal-05480691). Semantics and limits: docs/META_HARNESS_BELIEF_STATE.md.

Company OS Continual Harness slice — after each meta_harness.py candidate run, scripts/meta-harness/company_os_refiner_pass.py now normalizes live agent-chat metrics and NDJSON logs into candidate/refiner/company_os_turns.jsonl, then writes review-only company_os_harness_delta_<n>.json artifacts. Defaults: HSM_COMPANY_OS_REFINER_W=0, HSM_COMPANY_OS_REFINER_F=1 so short smoke runs still produce a delta candidate. These artifacts target prompt policy, skill dispatch, runtime guards, and memory/context policy, but they are not auto-applied.

Company eval packs — python3 scripts/meta-harness/meta_harness.py --pack path/to/company-eval-pack.yaml loads a declarative pack through scripts/meta-harness/pack_loader.py, sends each task through agent-chat, and passes pack KPI assertions into scripts/meta-harness/kpi_verifier.py. The versioned schema is schemas/company_eval_pack.v1.json; the field and scoring contract is documented in company-os/eval-pack-and-scoring-contract.md.

---

Continual Harness–style F-step loop on turns_hsm.jsonl

This is an offline, eval-side stub inspired by Continual Harness (arXiv:2605.09998 PDF), not a full implementation of the paper. In our current repo, one step = one JSONL record from turns_hsm.jsonl (same artifact family as hsm-eval --artifacts / hsm_meta_harness cand_*/ trees — see Where artifacts go above).

Paper implementation status

Paper mechanism	Current HSM-II status
Agent acts in one continuous environment while harness state updates in-place every F steps after warm-up W.	Partial / eval-only. refiner-jsonl-pass.py fires every F JSONL rows after W, but only after an eval trace exists; it does not update the live agent mid-run.
Refiner reads recent trajectory and emits edits to prompt (p), sub-agents (G), skills (K), and memory (M).	Stub. The script detects deterministic failure signatures and writes a harness_delta_<n>.json with placeholder sections for prompt/agent/skill/memory edits. No LLM Refiner currently fills or applies those edits.
Harness updates are applied immediately: H ← H ⊕ Δ.	Missing. Deltas are written for inspection; safe default is manual review + re-test with hsm-eval.
Failure signatures include loops, tool failures, stalls, and missed exploration/opportunities.	Partial. We detect repeated tool/no-tool patterns, repeated errors, repeated response stalls, simple task/turn stalls, and low rubric scores. Missed exploration is not modeled.
Reset-free model-harness co-learning with PRM scoring, frontier-teacher relabeling, and soft SFT.	Not implemented. The repo has eval scoring and an experimental belief-state logger for Company OS agent-chat, but no PRM → teacher relabel → SFT loop.
Embodied Pokemon environment, button-press cost, milestone metrics, pathfinding-skill oracle analysis.	Not implemented. HSM-II uses software/company/eval harness tasks, not the paper's Pokemon environment.

So: the current feature is best described as a trace-to-delta proving path for HSM-II eval artifacts. It verifies that we can observe behavior, write artifacts, detect basic patterns, and produce a structured improvement candidate. It does not yet provide online self-improving live agents in the paper's sense.

For Company OS agent-chat, the analogous proving path is:

meta_harness.py / evaluate_turn.py
  -> results.json + results/*.ndjson
  -> company_os_refiner_pass.py
  -> refiner/company_os_turns.jsonl
  -> refiner/company_os_harness_delta_<n>.json

That path is now implemented for live Company OS agent-chat observations. It remains review-only until a later integration safely maps approved deltas into the runtime prompt policy, skill dispatch, context/memory policy, or worker loop guards.

The current refiner artifact chooses one primary failure signature and emits at most one proposed change. When the source result includes its prompt and company identity, it also emits one trace/run-linked hsm.company_os.sdk_eval_experiment.v1 draft. The draft is not runnable until an operator adds expected-state or deterministic KPI assertions. The meta-harness converts the draft into eval-pack YAML for review. Other patch surfaces in the delta are explicitly deferred, so one failing window does not fan out into unrelated edits.

Filled deltas preserve the single-surface selection. A candidate-local proposal requires a passing paired runtime release report, candidate and baseline RuntimePolicy bytes that match the observed policy digests, a complete governed child-run report, and independent PR review evidence from two distinct reviewers. --approved-by is rejected; authenticated Company OS governance handles verification, human review, held-out/canary evidence, and activation. The review schema requires every finding to be closed with evidence or rejected with a reason and evidence. A fixed finding must include a passing test's argv command, exit code, captured output and SHA-256 digest, all bound to the reviewed PR head commit. Re-run review on the final tested head before preparing promotion evidence.

When fixing a finding changes the PR head, finalize that round with company_os_harness_review.py finalize --repo ., then rerun independent reviewers on the new HEAD with run --prior-review-report <previous-report>. The final report carries the earlier resolved findings, and the promotion gate checks that their tested commits are ancestors of the paired candidate head.

Trace-derived SDK experiments carry observed tool sequences and KPI evidence into the refiner prompt. In LLM mode, the refiner may propose expected_tools, an end-state, and deterministic state assertions only when those facts are supported by trace evidence; the proposal remains a draft for human curation. company_eval_experiment_to_pack.py turns expected tools into deterministic tool-sequence checks, and the voice-scenario converter refuses cases without an end-state rubric and deterministic KPI. Trace-derived SDK experiments can be converted to reviewed eval packs and then to scenario fixtures with semantic judge criteria and deterministic end-state KPIs. company_os_voice_eval_matrix.py runs each fixture through the cascaded Company OS path or configured speech-to-speech caller adapter, saving digest-bound traces, scorer evidence, and results. The paired gate compares every scenario across both runtimes and against the baseline. Provider-specific speech-to-speech transport is supplied through HSM_SPEECH_TO_SPEECH_CALLER_COMMAND.

company_os_harness_review.py run launches independent reviewer commands against the same frozen PR diff and candidate RuntimePolicy JSON. finalize refuses unresolved findings and emits the report checked by the review gate. company_os_promote_delta.py writes an inert local proposal. Submit it with company_os_promote_runtime_policy.py --harness-proposal --post to create a review-only RuntimePolicyDelta; the server's separate authenticated verifier, reviewer, held-out/canary, rollback, and activation gates remain required.

Symbol	Suggested value	Meaning
W	10	Warm-up lines: accumulate τ only; no Refiner (enough records for tool names / errors to appear).
F	50	Refiner runs every 50 lines after warm-up (lines 11–60 → refine #1, 61–110 → refine #2, …).

State: sliding window τ = last max(256, F) parsed JSONL objects for one run (ring buffer while streaming).

Refiner gate: after each appended line, let n = total lines read for this run. If n ≤ W, skip. If (n - W) mod F ≠ 0, skip. Else run one Refiner cycle.

Failure signatures (cheap predicates on the last F lines only):

• Loop: same tool_name (or same error substring) ≥ 3 times in the window.
• Tool fault: any line with error / non-success status / empty tool result where a payload was expected (map to your trace schema fields).
• Stall: no change in a coarse progress key (e.g. same suite/task_id subphase string) for ≥ half of F` lines when that key exists.
• Low quality: mean score < 0.5 over lines that carry score; if absent, use “assistant message with no tool calls” for 8 consecutive lines as a proxy.

Refiner output (HarnessDelta JSON) — four passes, targeting eval harness only (not live personal_agent):

1. Prompt / instructions (p): text patch for the runner instruction block or suite preamble (whatever your trace attributes carry).
2. Sub-agents / personas (𝒢): CRUD on named roles in HsmRunnerConfig or sidecar JSON the eval uses.
3. Skills / tools (𝒦): add/remove tool allowlist entries or skill refs recorded in the harness JSON.
4. Memory policy (𝒳): patch injection / recall limits / HsmRunnerConfig memory knobs reflected in traces.

Apply rule (safe default): write harness_delta_<n>.json next to the run; do not auto-promote. Re-verify with hsm-eval --hsm-config <patched> on a small --limit, then fold into hsm_meta_harness search or manual promote when the gate passes.

Runnable stub: scripts/refiner-jsonl-pass.py (deterministic signatures + harness_delta_<n>.json; no LLM). Tests: python3 scripts/test_refiner_jsonl_pass.py.

Live path (LLM + hsm-eval + refiner): bash scripts/refiner-hsm-eval-smoke.sh — runs hsm-eval --suite memory --limit 1 --artifacts … (verbose), finds **/turns_hsm.jsonl, then runs the refiner with small HSM_REFINER_W / HSM_REFINER_F (default 2/2) so short traces still emit at least one harness_delta_*.json. Requires OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY, or Ollama on PATH. Override output dir with HSM_REFINER_EVAL_OUT.

---

Related references

• META_HARNESS_BELIEF_STATE.md — Beta / VoI-proxy fields next to score in evaluate_turn.py.
• GOLDEN_PATH.md — Ladybug path; includes quick hsm-eval suite commands.
• documentation/guides/HARNESS_V1_PLAN.md — harness hardening plan.
• templates/business/starters/online_commerce_squad/knowledge/dspy_gepa_hsm_bridge.md — DSPy/GEPA and meta-harness in the “improve over time” story.
• README — “Other ways to run” → external harnesses and YC-bench via hsm_outer_loop.