Eval, meta-harness, and outer loop

This document is the canonical guide for when to use each tool, where artifacts land, and the contract between promoted harness JSON and the rest of HSM-II.

For Company OS agent-chat regression (locked NEXT_URL / HSM_URL, meta-harness vs LongMemEval, optional services/agent-chat-harness), see company-os/AGENT_CHAT_REFERENCE_HARNESS_PROFILE.md.

For cross-company declarative eval packs, YAML schema, KPI scorer bindings, and the Company OS agent-chat scoring/promotion contract, see company-os/eval-pack-and-scoring-contract.md.

For standardized academic model-quality benchmarks through EleutherAI lm-evaluation-harness, see company-os/lm-evaluation-harness-lane.md.

FlashEvolve (async evolution orchestration) — arXiv:2605.08520 targets wall-clock cost of GEPA/Meta-Harness loops via queued overlapping stages. HSM-II runs those stages synchronously today; FlashEvolve is the planned throughput layer. See company-os/ax-harness-evolution-stack.md.

For native SMB-style benchmarks (different harness), see HSM_NATIVE_BENCH.md.

Which tool when

Tool	Role	Use it when
`hsm-eval`	Inner eval runtime — runs the benchmark suite with `HsmRunner` vs a vanilla LLM baseline, writes metrics and optional JSONL traces.	You want a single measurement pass, a fixed `HsmRunnerConfig`, or to validate a JSON harness file (e.g. after meta-harness) with `--hsm-config`.
`hsm_meta_harness`	Outer search (phase 1) — samples or loads many `HsmRunnerConfig` candidates, compares each to baseline, ranks by objective, exports Pareto frontier, optionally writes `best_config.json` and can promote a file to `config/hsm_harness.default.json`.	You want to search harness knobs (memory injection, budgets, thresholds, etc.) against the same verifiable tasks, not tweak weights.
`hsm_outer_loop`	Outer-loop infrastructure — compile gate, SQLite over `runs/runs_index.jsonl`, queries, proposer context for agents, external Rust benchmark batches (e.g. YC-bench, side repos).	You need to index/query past runs, feed a coding agent context from history, or run non-in-tree harnesses from JSON specs.
`scripts/lm-eval-company-os.sh`	Academic model-quality lane — wraps EleutherAI `lm_eval` presets and writes artifacts under `runs/lm-eval/`.	You want standardized MMLU/GSM8K/HellaSwag/TruthfulQA-style scores for model choice or release reporting.

Mental model: hsm-eval = one HSM-native experiment; hsm_meta_harness = many candidates + leaderboard; hsm_outer_loop = archive/DB/tooling around runs; lm_eval = standardized academic model quality, not Company OS workflow quality.

Contract: promoted config vs production runtime

HsmRunnerConfig is the tunable policy object used by HsmRunner inside hsm-eval, hsm_meta_harness, and related eval binaries (hsm_native_eval, hsm_longmemeval, etc.).
personal_agent, hsm_console, and the main Telegram/API agent stack do not automatically load config/hsm_harness.default.json or HSM_META_HARNESS_CONFIG. Their memory and tool behavior use different configuration paths.
Therefore: meta-harness and hsm-eval are eval-side tooling. A promoted best_config.json is mapped to live Postgres policy via POST .../runtime-policy/import-best-config (active runtime_policy_versions; agent-chat + worker read GET .../runtime-policy/active). A promoted best_config.json (or copied config/hsm_harness.default.json) is not wired into the live bot until an explicit integration maps those fields into the runtime you run in production.

Practical implication: Treat best_config.json as the artifact you re-test with hsm-eval --hsm-config path/to/best_config.json and, if you change product behavior manually, as documentation of what worked on benchmarks — not as a switch that flips personal_agent today.

Where artifacts go

Default layout uses a runs/ directory at the repo root (override with --out-dir / --artifacts where supported).

`hsm-eval`

With --artifacts <dir>: writes manifest.json, comparison outputs, and per-suite dirs with turns_hsm.jsonl, turns_baseline.jsonl, optional hsm_trace.jsonl, paths recorded under artifact_paths in the manifest.
May append one line to runs/runs_index.jsonl (unless disabled) for outer-loop ingestion.

`hsm_meta_harness`

Default run directory: runs/run_<unix_timestamp>/ (or --out-dir).
Per run:
- baseline_by_suite.json (and baseline_metrics.json if a single suite).
- Under cand_*/: candidate_result.json, per_suite.json, per-suite subdirs with hsm_metrics.json, comparison_report.json, turns_hsm.jsonl, turns_baseline.jsonl, optional hsm_trace.jsonl.
- leaderboard.json, pareto_frontier.json, manifest.json.
- If the confidence gate passes: best_config.json (full HsmRunnerConfig JSON).
promote subcommand copies a harness JSON to config/hsm_harness.default.json by default (see hsm_meta_harness promote --help).

`hsm_outer_loop`

external-batch / external: write results under paths given in the spec (e.g. runs/external_batch_<timestamp>.json).
index-db: builds runs/runs.sqlite from runs/runs_index.jsonl (paths configurable).
propose: emits ProposerContext JSON (e.g. for agent workflows).

Environment variables (see .env.example): HSM_RUNS_SQLITE, HSM_PARENT_RUN_ID, HSM_META_HARNESS_CONFIG, eval thresholds, etc.

Smoke recipe (copy-paste)

Prerequisites: Rust toolchain, repo clone, and any one LLM path the rest of the project uses (e.g. Ollama running locally, or OPENAI_API_KEY / ANTHROPIC_API_KEY set — same as for hsm-eval).

Minimal Ollama example:

bash

cd /path/to/HSM-II
# Ensure your model is available, e.g.:
# ollama pull llama3.2
export OLLAMA_MODEL=llama3.2

1) Single eval (hsm-eval) — tiny slice

bash

cargo run --bin hsm-eval -- --suite memory --limit 2 --verbose

2) Eval with artifacts (for Trace2Skill / inspection)

bash

mkdir -p runs/smoke_eval
cargo run --bin hsm-eval -- --suite memory --limit 2 --artifacts runs/smoke_eval

3) Meta-harness — smoke search (small sample)

Meta-harness enforces a minimum task count unless you opt out:

bash

cargo run --bin hsm_meta_harness -- \
  --candidates 2 \
  --bootstrap-runs 1 \
  --suite memory \
  --limit 2 \
  --allow-small-sample \
  --require-positive-ci=false

4) Outer loop — list runs

After a run that appended runs/runs_index.jsonl:

bash

cargo run --bin hsm_outer_loop -- list-runs --index runs/runs_index.jsonl --limit 10

5) Validate a promoted or best_config.json with hsm-eval

bash

cargo run --bin hsm-eval -- \
  --suite memory \
  --limit 2 \
  --hsm-config runs/run_<timestamp>/best_config.json

(Use the actual path to your best_config.json.)

Company OS agent-chat (Python `scripts/meta-harness`)

This is a separate harness from the Rust hsm_meta_harness binary: it drives the live Next.js Company Console routes POST /api/agent-chat-reply/stream and POST /api/agent-chat-reply, talking to hsm_console for tasks and worker execution (evaluate_turn.py defaults: Next http://127.0.0.1:3050, HSM http://127.0.0.1:3847).

Prerequisites

cargo run -p hyper-stigmergy --bin hsm_console (or your usual Company OS API) reachable on 3847 (or override in evaluate_turn.py).
cd web/company-console && npm run dev (or production build) on 3050 so the stream route exists.
LLM keys the console expects (e.g. OpenRouter / OPENROUTER_API_KEY per web/company-console docs).
Harness data directory: by default the script uses ~/.hsm/meta-harness/ when your home directory is writable. If not (e.g. Cursor sandbox, read-only CI), it automatically falls back to <repo>/.meta-harness/ (gitignored). Override anytime with HSM_META_HARNESS_DATA_DIR.

Timeouts

Server-side worker telemetry waits are controlled by HSM_OPERATOR_CHAT_TELEMETRY_WAIT_EXEC_MS / HSM_OPERATOR_CHAT_TELEMETRY_WAIT_ANALYSIS_MS (see .env.example). JSON and NDJSON paths share the same caps; build-heavy skills get a longer minimum wait.
Next route maxDuration and the browser fallback poll budget are defined in web/company-console/app/lib/operator-chat-timeouts.ts so hosted and UI behavior stay above those waits.

Commands

bash

cd /path/to/hyper-stigmergic-morphogenesisII
# Quick smoke (default: `--paper-smoke` — three tasks aligned to Meta-Harness paper harness dimensions: tools + synthesis + memory recall; legacy: `MH_SMOKE_LEGACY=1` uses first N tasks)
./scripts/company-os-agent-chat-meta-harness-smoke.sh

# Same paper-aligned trio, invoked directly
python3 scripts/meta-harness/meta_harness.py --paper-smoke

# First three rows of `EVAL_TASKS` only (all engineering prompts)
python3 scripts/meta-harness/meta_harness.py --tasks 3

Belief state (experimental) — evaluate_turn.py also prints a belief_state object: a Beta-style summary over a single “task success” latent plus a VoI proxy for logging, aligned with Papamarkou et al. (Bayes-consistent decisions; HAL hal-05480691). Semantics and limits: docs/META_HARNESS_BELIEF_STATE.md.

Company OS Continual Harness slice — after each meta_harness.py candidate run, scripts/meta-harness/company_os_refiner_pass.py now normalizes live agent-chat metrics and NDJSON logs into candidate/refiner/company_os_turns.jsonl, then writes review-only company_os_harness_delta_<n>.json artifacts. Defaults: HSM_COMPANY_OS_REFINER_W=0, HSM_COMPANY_OS_REFINER_F=1 so short smoke runs still produce a delta candidate. These artifacts target prompt policy, skill dispatch, runtime guards, and memory/context policy, but they are not auto-applied.

Company eval packs — python3 scripts/meta-harness/meta_harness.py --pack path/to/company-eval-pack.yaml loads a declarative pack through scripts/meta-harness/pack_loader.py, sends each task through agent-chat, and passes pack KPI assertions into scripts/meta-harness/kpi_verifier.py. The versioned schema is schemas/company_eval_pack.v1.json; the field and scoring contract is documented in company-os/eval-pack-and-scoring-contract.md.

Continual Harness–style F-step loop on `turns_hsm.jsonl`

This is an offline, eval-side stub inspired by Continual Harness (arXiv:2605.09998 PDF), not a full implementation of the paper. In our current repo, one step = one JSONL record from turns_hsm.jsonl (same artifact family as hsm-eval --artifacts / hsm_meta_harness cand_*/ trees — see Where artifacts go above).

Paper implementation status

Paper mechanism	Current HSM-II status
Agent acts in one continuous environment while harness state updates in-place every F steps after warm-up W.	Partial / eval-only. `refiner-jsonl-pass.py` fires every F JSONL rows after W, but only after an eval trace exists; it does not update the live agent mid-run.
Refiner reads recent trajectory and emits edits to prompt (`p`), sub-agents (`G`), skills (`K`), and memory (`M`).	Stub. The script detects deterministic failure signatures and writes a `harness_delta_<n>.json` with placeholder sections for prompt/agent/skill/memory edits. No LLM Refiner currently fills or applies those edits.
Harness updates are applied immediately: `H ← H ⊕ Δ`.	Missing. Deltas are written for inspection; safe default is manual review + re-test with `hsm-eval`.
Failure signatures include loops, tool failures, stalls, and missed exploration/opportunities.	Partial. We detect repeated tool/no-tool patterns, repeated errors, repeated response stalls, simple task/turn stalls, and low rubric scores. Missed exploration is not modeled.
Reset-free model-harness co-learning with PRM scoring, frontier-teacher relabeling, and soft SFT.	Not implemented. The repo has eval scoring and an experimental belief-state logger for Company OS agent-chat, but no PRM → teacher relabel → SFT loop.
Embodied Pokemon environment, button-press cost, milestone metrics, pathfinding-skill oracle analysis.	Not implemented. HSM-II uses software/company/eval harness tasks, not the paper's Pokemon environment.

So: the current feature is best described as a trace-to-delta proving path for HSM-II eval artifacts. It verifies that we can observe behavior, write artifacts, detect basic patterns, and produce a structured improvement candidate. It does not yet provide online self-improving live agents in the paper's sense.

For Company OS agent-chat, the analogous proving path is:

text

meta_harness.py / evaluate_turn.py
  -> results.json + results/*.ndjson
  -> company_os_refiner_pass.py
  -> refiner/company_os_turns.jsonl
  -> refiner/company_os_harness_delta_<n>.json

That path is now implemented for live Company OS agent-chat observations. It remains review-only until a later integration safely maps approved deltas into the runtime prompt policy, skill dispatch, context/memory policy, or worker loop guards.

Symbol	Suggested value	Meaning
W	10	Warm-up lines: accumulate τ only; no Refiner (enough records for tool names / errors to appear).
F	50	Refiner runs every 50 lines after warm-up (lines 11–60 → refine #1, 61–110 → refine #2, …).

State: sliding window τ = last max(256, F) parsed JSONL objects for one run (ring buffer while streaming).

Refiner gate: after each appended line, let n = total lines read for this run. If n ≤ W, skip. If (n - W) mod F ≠ 0, skip. Else run one Refiner cycle.

Failure signatures (cheap predicates on the last F lines only):

Loop: same tool_name (or same error substring) ≥ 3 times in the window.
Tool fault: any line with error / non-success status / empty tool result where a payload was expected (map to your trace schema fields).
Stall: no change in a coarse progress key (e.g. same suite/task_id subphase string) for ≥ half of F` lines when that key exists.
Low quality: mean score < 0.5 over lines that carry score; if absent, use “assistant message with no tool calls” for 8 consecutive lines as a proxy.

Refiner output (HarnessDelta JSON) — four passes, targeting eval harness only (not live personal_agent):

Prompt / instructions (p): text patch for the runner instruction block or suite preamble (whatever your trace attributes carry).
Sub-agents / personas (𝒢): CRUD on named roles in HsmRunnerConfig or sidecar JSON the eval uses.
Skills / tools (𝒦): add/remove tool allowlist entries or skill refs recorded in the harness JSON.
Memory policy (𝒳): patch injection / recall limits / HsmRunnerConfig memory knobs reflected in traces.

Apply rule (safe default): write harness_delta_<n>.json next to the run; do not auto-promote. Re-verify with hsm-eval --hsm-config <patched> on a small --limit, then fold into hsm_meta_harness search or manual promote when the gate passes.

Runnable stub: scripts/refiner-jsonl-pass.py (deterministic signatures + harness_delta_<n>.json; no LLM). Tests: python3 scripts/test_refiner_jsonl_pass.py.

Live path (LLM + hsm-eval + refiner): bash scripts/refiner-hsm-eval-smoke.sh — runs hsm-eval --suite memory --limit 1 --artifacts … (verbose), finds **/turns_hsm.jsonl, then runs the refiner with small HSM_REFINER_W / HSM_REFINER_F (default 2/2) so short traces still emit at least one harness_delta_*.json. Requires OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY, or Ollama on PATH. Override output dir with HSM_REFINER_EVAL_OUT.

META_HARNESS_BELIEF_STATE.md — Beta / VoI-proxy fields next to score in evaluate_turn.py.
GOLDEN_PATH.md — Ladybug path; includes quick hsm-eval suite commands.
documentation/guides/HARNESS_V1_PLAN.md — harness hardening plan.
templates/business/starters/online_commerce_squad/knowledge/dspy_gepa_hsm_bridge.md — DSPy/GEPA and meta-harness in the “improve over time” story.
README — “Other ways to run” → external harnesses and YC-bench via hsm_outer_loop.

Eval, meta-harness, and outer loop ​

Which tool when ​

Contract: promoted config vs production runtime ​

Where artifacts go ​

hsm-eval ​

hsm_meta_harness ​

hsm_outer_loop ​

Smoke recipe (copy-paste) ​

Company OS agent-chat (Python scripts/meta-harness) ​

Continual Harness–style F-step loop on turns_hsm.jsonl ​

Paper implementation status ​

Related references ​

Eval, meta-harness, and outer loop

Which tool when

Contract: promoted config vs production runtime

Where artifacts go

`hsm-eval`

`hsm_meta_harness`

`hsm_outer_loop`

Smoke recipe (copy-paste)

Company OS agent-chat (Python `scripts/meta-harness`)

Continual Harness–style F-step loop on `turns_hsm.jsonl`

Paper implementation status

Related references