# Eval, meta-harness, and outer loop

This document is the **canonical** guide for when to use each tool, **where artifacts land**, and the **contract** between promoted harness JSON and the rest of HSM-II.

For **Company OS agent-chat** regression (locked **`NEXT_URL` / `HSM_URL`**, meta-harness vs LongMemEval, optional **`services/agent-chat-harness`**), see [`company-os/AGENT_CHAT_REFERENCE_HARNESS_PROFILE.md`](./company-os/AGENT_CHAT_REFERENCE_HARNESS_PROFILE.md).

For cross-company declarative eval packs, YAML schema, KPI scorer bindings, and the Company OS agent-chat scoring/promotion contract, see [`company-os/eval-pack-and-scoring-contract.md`](./company-os/eval-pack-and-scoring-contract.md).

For standardized academic model-quality benchmarks through EleutherAI
`lm-evaluation-harness`, see
[`company-os/lm-evaluation-harness-lane.md`](./company-os/lm-evaluation-harness-lane.md).

**FlashEvolve (async evolution orchestration)** — [arXiv:2605.08520](https://arxiv.org/pdf/2605.08520) targets wall-clock cost of GEPA/Meta-Harness loops via queued overlapping stages. HSM-II runs those stages synchronously today; FlashEvolve is the planned throughput layer. See [`company-os/ax-harness-evolution-stack.md`](./company-os/ax-harness-evolution-stack.md).

For native SMB-style benchmarks (different harness), see [`HSM_NATIVE_BENCH.md`](./HSM_NATIVE_BENCH.md).

---

## Which tool when

| Tool | Role | Use it when |
|------|------|-------------|
| **`hsm-eval`** | **Inner eval runtime** — runs the benchmark suite with `HsmRunner` vs a vanilla LLM baseline, writes metrics and optional JSONL traces. | You want a **single** measurement pass, a fixed `HsmRunnerConfig`, or to **validate** a JSON harness file (e.g. after meta-harness) with `--hsm-config`. |
| **`hsm_meta_harness`** | **Outer search (phase 1)** — samples or loads **many** `HsmRunnerConfig` candidates, compares each to baseline, ranks by objective, exports **Pareto frontier**, optionally writes **`best_config.json`** and can **promote** a file to `config/hsm_harness.default.json`. | You want to **search** harness knobs (memory injection, budgets, thresholds, etc.) against the same verifiable tasks, not tweak weights. |
| **`hsm_outer_loop`** | **Outer-loop infrastructure** — compile gate, **SQLite** over `runs/runs_index.jsonl`, **queries**, **proposer** context for agents, **external** Rust benchmark batches (e.g. YC-bench, side repos). | You need to **index/query** past runs, feed a coding agent **context** from history, or run **non-in-tree** harnesses from JSON specs. |
| **`scripts/lm-eval-company-os.sh`** | **Academic model-quality lane** — wraps EleutherAI `lm_eval` presets and writes artifacts under `runs/lm-eval/`. | You want standardized MMLU/GSM8K/HellaSwag/TruthfulQA-style scores for model choice or release reporting. |

**Mental model:** `hsm-eval` = one HSM-native experiment; `hsm_meta_harness` = many candidates + leaderboard; `hsm_outer_loop` = archive/DB/tooling around runs; `lm_eval` = standardized academic model quality, not Company OS workflow quality.

---

## Contract: promoted config vs production runtime

- **`HsmRunnerConfig`** is the tunable policy object used by **`HsmRunner`** inside **`hsm-eval`**, **`hsm_meta_harness`**, and related eval binaries (`hsm_native_eval`, `hsm_longmemeval`, etc.).
- **`personal_agent`**, **`hsm_console`**, and the main Telegram/API agent stack **do not** automatically load `config/hsm_harness.default.json` or `HSM_META_HARNESS_CONFIG`. Their memory and tool behavior use **different** configuration paths.
- Therefore: **meta-harness and `hsm-eval` are eval-side tooling.** A promoted **`best_config.json`** is mapped to live Postgres policy via **`POST .../runtime-policy/import-best-config`** (active `runtime_policy_versions`; agent-chat + worker read `GET .../runtime-policy/active`). A promoted **`best_config.json`** (or copied **`config/hsm_harness.default.json`**) is **not** wired into the live bot until an explicit integration maps those fields into the runtime you run in production.

**Practical implication:** Treat **`best_config.json`** as the artifact you **re-test** with `hsm-eval --hsm-config path/to/best_config.json` and, if you change product behavior manually, as documentation of what worked on benchmarks — not as a switch that flips `personal_agent` today.

---

## Where artifacts go

Default layout uses a **`runs/`** directory at the repo root (override with `--out-dir` / `--artifacts` where supported).

### `hsm-eval`

- With **`--artifacts <dir>`**: writes **`manifest.json`**, comparison outputs, and per-suite dirs with **`turns_hsm.jsonl`**, **`turns_baseline.jsonl`**, optional **`hsm_trace.jsonl`**, paths recorded under **`artifact_paths`** in the manifest.
- May append one line to **`runs/runs_index.jsonl`** (unless disabled) for outer-loop ingestion.

### `hsm_meta_harness`

- Default run directory: **`runs/run_<unix_timestamp>/`** (or **`--out-dir`**).
- Per run:
  - **`baseline_by_suite.json`** (and **`baseline_metrics.json`** if a single suite).
  - Under **`cand_*/`**: **`candidate_result.json`**, **`per_suite.json`**, per-suite subdirs with **`hsm_metrics.json`**, **`comparison_report.json`**, **`turns_hsm.jsonl`**, **`turns_baseline.jsonl`**, optional **`hsm_trace.jsonl`**.
  - **`leaderboard.json`**, **`pareto_frontier.json`**, **`manifest.json`**.
  - If the confidence gate passes: **`best_config.json`** (full **`HsmRunnerConfig`** JSON).
- **`promote` subcommand** copies a harness JSON to **`config/hsm_harness.default.json`** by default (see `hsm_meta_harness promote --help`).

### `hsm_outer_loop`

- **`external-batch`** / **`external`**: write results under paths given in the spec (e.g. **`runs/external_batch_<timestamp>.json`**).
- **`index-db`**: builds **`runs/runs.sqlite`** from **`runs/runs_index.jsonl`** (paths configurable).
- **`propose`**: emits **ProposerContext** JSON (e.g. for agent workflows).

Environment variables (see **`.env.example`**): `HSM_RUNS_SQLITE`, `HSM_PARENT_RUN_ID`, `HSM_META_HARNESS_CONFIG`, eval thresholds, etc.

---

## Smoke recipe (copy-paste)

**Prerequisites:** Rust toolchain, repo clone, and **any one** LLM path the rest of the project uses (e.g. Ollama running locally, or `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` set — same as for `hsm-eval`).

Minimal Ollama example:

```bash
cd /path/to/HSM-II
# Ensure your model is available, e.g.:
# ollama pull llama3.2
export OLLAMA_MODEL=llama3.2
```

**1) Single eval (`hsm-eval`) — tiny slice**

```bash
cargo run --bin hsm-eval -- --suite memory --limit 2 --verbose
```

**2) Eval with artifacts (for Trace2Skill / inspection)**

```bash
mkdir -p runs/smoke_eval
cargo run --bin hsm-eval -- --suite memory --limit 2 --artifacts runs/smoke_eval
```

**3) Meta-harness — smoke search (small sample)**

Meta-harness enforces a minimum task count unless you opt out:

```bash
cargo run --bin hsm_meta_harness -- \
  --candidates 2 \
  --bootstrap-runs 1 \
  --suite memory \
  --limit 2 \
  --allow-small-sample \
  --require-positive-ci=false
```

**4) Outer loop — list runs**

After a run that appended **`runs/runs_index.jsonl`**:

```bash
cargo run --bin hsm_outer_loop -- list-runs --index runs/runs_index.jsonl --limit 10
```

**5) Validate a promoted or `best_config.json` with `hsm-eval`**

```bash
cargo run --bin hsm-eval -- \
  --suite memory \
  --limit 2 \
  --hsm-config runs/run_<timestamp>/best_config.json
```

(Use the actual path to your `best_config.json`.)

---

## Company OS agent-chat (Python `scripts/meta-harness`)

This is a **separate** harness from the Rust `hsm_meta_harness` binary: it drives the live **Next.js** Company Console routes `POST /api/agent-chat-reply/stream` and `POST /api/agent-chat-reply`, talking to **`hsm_console`** for tasks and worker execution (`evaluate_turn.py` defaults: Next `http://127.0.0.1:3050`, HSM `http://127.0.0.1:3847`).

**Prerequisites**

- `cargo run -p hyper-stigmergy --bin hsm_console` (or your usual Company OS API) reachable on **3847** (or override in `evaluate_turn.py`).
- `cd web/company-console && npm run dev` (or production build) on **3050** so the stream route exists.
- LLM keys the console expects (e.g. **OpenRouter** / `OPENROUTER_API_KEY` per `web/company-console` docs).
- **Harness data directory:** by default the script uses **`~/.hsm/meta-harness/`** when your home directory is writable. If not (e.g. Cursor sandbox, read-only CI), it automatically falls back to **`<repo>/.meta-harness/`** (gitignored). Override anytime with **`HSM_META_HARNESS_DATA_DIR`**.

**Timeouts**

- Server-side worker telemetry waits are controlled by **`HSM_OPERATOR_CHAT_TELEMETRY_WAIT_EXEC_MS`** / **`HSM_OPERATOR_CHAT_TELEMETRY_WAIT_ANALYSIS_MS`** (see **`.env.example`**). JSON and NDJSON paths share the same caps; build-heavy skills get a longer minimum wait.
- Next route **`maxDuration`** and the browser fallback poll budget are defined in **`web/company-console/app/lib/operator-chat-timeouts.ts`** so hosted and UI behavior stay above those waits.

**Commands**

```bash
cd /path/to/hyper-stigmergic-morphogenesisII
# Quick smoke (default: `--paper-smoke` — three tasks aligned to Meta-Harness paper harness dimensions: tools + synthesis + memory recall; legacy: `MH_SMOKE_LEGACY=1` uses first N tasks)
./scripts/company-os-agent-chat-meta-harness-smoke.sh

# Same paper-aligned trio, invoked directly
python3 scripts/meta-harness/meta_harness.py --paper-smoke

# First three rows of `EVAL_TASKS` only (all engineering prompts)
python3 scripts/meta-harness/meta_harness.py --tasks 3
```

**Belief state (experimental)** — `evaluate_turn.py` also prints a **`belief_state`** object: a Beta-style summary over a single “task success” latent plus a **VoI proxy** for logging, aligned with Papamarkou et al. (*Bayes-consistent decisions*; [HAL hal-05480691](https://hal.science/hal-05480691)). Semantics and limits: **`docs/META_HARNESS_BELIEF_STATE.md`**.

**Company OS Continual Harness slice** — after each `meta_harness.py` candidate run, **`scripts/meta-harness/company_os_refiner_pass.py`** now normalizes live agent-chat metrics and NDJSON logs into **`candidate/refiner/company_os_turns.jsonl`**, then writes review-only **`company_os_harness_delta_<n>.json`** artifacts. Defaults: **`HSM_COMPANY_OS_REFINER_W=0`**, **`HSM_COMPANY_OS_REFINER_F=1`** so short smoke runs still produce a delta candidate. These artifacts target prompt policy, skill dispatch, runtime guards, and memory/context policy, but they are **not auto-applied**.

**Company eval packs** — `python3 scripts/meta-harness/meta_harness.py --pack path/to/company-eval-pack.yaml` loads a declarative pack through **`scripts/meta-harness/pack_loader.py`**, sends each task through agent-chat, and passes pack KPI assertions into **`scripts/meta-harness/kpi_verifier.py`**. The versioned schema is **`schemas/company_eval_pack.v1.json`**; the field and scoring contract is documented in **[`company-os/eval-pack-and-scoring-contract.md`](./company-os/eval-pack-and-scoring-contract.md)**.

---

## Continual Harness–style **F-step** loop on **`turns_hsm.jsonl`**

This is an **offline, eval-side stub inspired by** Continual Harness ([arXiv:2605.09998 PDF](https://arxiv.org/pdf/2605.09998)), not a full implementation of the paper. In our current repo, **one step = one JSONL record** from **`turns_hsm.jsonl`** (same artifact family as **`hsm-eval --artifacts`** / **`hsm_meta_harness`** `cand_*/` trees — see **Where artifacts go** above).

### Paper implementation status

| Paper mechanism | Current HSM-II status |
|-----------------|-----------------------|
| Agent acts in one continuous environment while harness state updates in-place every **F** steps after warm-up **W**. | **Partial / eval-only.** `refiner-jsonl-pass.py` fires every **F** JSONL rows after **W**, but only after an eval trace exists; it does not update the live agent mid-run. |
| Refiner reads recent trajectory and emits edits to **prompt (`p`)**, **sub-agents (`G`)**, **skills (`K`)**, and **memory (`M`)**. | **Stub.** The script detects deterministic failure signatures and writes a `harness_delta_<n>.json` with placeholder sections for prompt/agent/skill/memory edits. No LLM Refiner currently fills or applies those edits. |
| Harness updates are applied immediately: `H ← H ⊕ Δ`. | **Missing.** Deltas are written for inspection; safe default is manual review + re-test with `hsm-eval`. |
| Failure signatures include loops, tool failures, stalls, and missed exploration/opportunities. | **Partial.** We detect repeated tool/no-tool patterns, repeated errors, repeated response stalls, simple task/turn stalls, and low rubric scores. Missed exploration is not modeled. |
| Reset-free model-harness co-learning with PRM scoring, frontier-teacher relabeling, and soft SFT. | **Not implemented.** The repo has eval scoring and an experimental belief-state logger for Company OS agent-chat, but no PRM → teacher relabel → SFT loop. |
| Embodied Pokemon environment, button-press cost, milestone metrics, pathfinding-skill oracle analysis. | **Not implemented.** HSM-II uses software/company/eval harness tasks, not the paper's Pokemon environment. |

So: the current feature is best described as a **trace-to-delta proving path** for HSM-II eval artifacts. It verifies that we can observe behavior, write artifacts, detect basic patterns, and produce a structured improvement candidate. It does **not** yet provide online self-improving live agents in the paper's sense.

For **Company OS agent-chat**, the analogous proving path is:

```text
meta_harness.py / evaluate_turn.py
  -> results.json + results/*.ndjson
  -> company_os_refiner_pass.py
  -> refiner/company_os_turns.jsonl
  -> refiner/company_os_harness_delta_<n>.json
```

That path is now implemented for live Company OS agent-chat observations. It remains **review-only** until a later integration safely maps approved deltas into the runtime prompt policy, skill dispatch, context/memory policy, or worker loop guards.

| Symbol | Suggested value | Meaning |
|--------|-----------------|--------|
| **W** | **10** | Warm-up lines: accumulate τ only; **no** Refiner (enough records for tool names / errors to appear). |
| **F** | **50** | Refiner runs every **50** lines **after** warm-up (lines 11–60 → refine #1, 61–110 → refine #2, …). |

**State:** sliding window **`τ`** = last **max(256, F)** parsed JSONL objects for one run (ring buffer while streaming).

**Refiner gate:** after each appended line, let **`n`** = total lines read for this run. If **`n ≤ W`**, skip. If **`(n - W) mod F ≠ 0`**, skip. Else run **one Refiner cycle**.

**Failure signatures** (cheap predicates on the **last F** lines only):

- **Loop:** same **`tool_name`** (or same error substring) ≥ **3** times in the window.
- **Tool fault:** any line with `error` / non-success status / empty tool result where a payload was expected (map to your trace schema fields).
- **Stall:** no change in a coarse progress key (e.g. same `suite`/`task_id` subphase string) for ≥ **half of F`** lines when that key exists.
- **Low quality:** mean **`score`** < **0.5** over lines that carry **`score`**; if absent, use “assistant message with no tool calls” for **8** consecutive lines as a proxy.

**Refiner output (`HarnessDelta` JSON)** — four passes, targeting **eval harness** only (not live `personal_agent`):

1. **Prompt / instructions** (`p`): text patch for the **runner instruction block** or suite preamble (whatever your trace attributes carry).
2. **Sub-agents / personas** (`𝒢`): CRUD on **named roles** in **`HsmRunnerConfig`** or sidecar JSON the eval uses.
3. **Skills / tools** (`𝒦`): add/remove **tool allowlist** entries or **skill refs** recorded in the harness JSON.
4. **Memory policy** (`𝒳`): patch **injection / recall limits** / `HsmRunnerConfig` memory knobs reflected in traces.

**Apply rule (safe default):** write **`harness_delta_<n>.json`** next to the run; **do not** auto-`promote`. Re-verify with **`hsm-eval --hsm-config <patched>`** on a **small `--limit`**, then fold into **`hsm_meta_harness`** search or manual **`promote`** when the gate passes.

**Runnable stub:** `scripts/refiner-jsonl-pass.py` (deterministic signatures + `harness_delta_<n>.json`; no LLM). **Tests:** `python3 scripts/test_refiner_jsonl_pass.py`.

**Live path (LLM + `hsm-eval` + refiner):** `bash scripts/refiner-hsm-eval-smoke.sh` — runs `hsm-eval --suite memory --limit 1 --artifacts …` (verbose), finds `**/turns_hsm.jsonl`, then runs the refiner with small **`HSM_REFINER_W` / `HSM_REFINER_F`** (default `2`/`2`) so short traces still emit at least one `harness_delta_*.json`. Requires **`OPENAI_API_KEY`**, **`ANTHROPIC_API_KEY`**, **`OPENROUTER_API_KEY`**, or **Ollama** on `PATH`. Override output dir with **`HSM_REFINER_EVAL_OUT`**.

---

## Related references

- **`META_HARNESS_BELIEF_STATE.md`** — Beta / VoI-proxy fields next to **`score`** in **`evaluate_turn.py`**.
- **`GOLDEN_PATH.md`** — Ladybug path; includes quick **`hsm-eval`** suite commands.
- **`documentation/guides/HARNESS_V1_PLAN.md`** — harness hardening plan.
- **`templates/business/starters/online_commerce_squad/knowledge/dspy_gepa_hsm_bridge.md`** — DSPy/GEPA and meta-harness in the “improve over time” story.
- **README** — “Other ways to run” → external harnesses and YC-bench via **`hsm_outer_loop`**.