# Reference harness profile (agent-chat + meta-harness + LongMemEval)

This is the **single stack** to use when you want **comparable** regression numbers across machines and models. Change these env vars and you are **not** doing an apples-to-apples run.

## Two gates (do not conflate them)

| Gate | What it measures | Primary commands / surfaces |
|------|------------------|-----------------------------|
| **Meta-harness** | **Worker stream health** — NDJSON from **`POST /api/agent-chat-reply/stream`**: phases, **`runtime`** / **`runtime_raw`** (`CompletionEvent`-shaped tool telemetry), **`done`** with **`reply`** + **`finalized`**, errors, composite score | `scripts/company-os-agent-chat-meta-harness-smoke.sh`, `scripts/meta-harness/evaluate_turn.py`, `scripts/meta-harness/meta_harness.py` |
| **LongMemEval** | **Memory QA accuracy** — oracle match / temporal reasoning vs haystack JSON | `cargo run --bin hsm-longmemeval …`, `cargo run --bin hsm-longmemeval-agent-chat …` (Rust API parity flags); optional **`hsm-eval`** / **`HsmRunner`** tuning |

High meta-harness scores usually require **`forceWorkerDispatch`** so turns hit **`operator-chat` → execute-worker** (tool counts + runtime lines). LongMemEval cares about **answers**, not NDJSON tool density.

## Locked URLs (defaults)

| Env | Default | Meaning |
|-----|-----------|---------|
| **`NEXT_URL`** | `http://127.0.0.1:3050` | Company Console (Next) — meta-harness reads this |
| **`HSM_URL`** / **`HSM_CONSOLE_URL`** | `http://127.0.0.1:3847` | Company OS (`hsm_console`) |
| **`HSM_COMPANY_CONSOLE_URL`** | same idea | Next origin for **`services/agent-chat-harness`** HTTP driver |

Meta-harness Python defaults align with **`evaluate_turn.py`** (`NEXT_URL`, `HSM_URL`).

## Operator chat / LLM (Next server-side)

Set these on the **Company Console** process (the Next app), not only in the shell running smoke tests:

| Env | Role |
|-----|------|
| **`OPENROUTER_API_KEY`** | Conversational path + many smoke configs |
| **`HSM_OPENROUTER_API_KEY`** | Alternate name picked up by Next |
| **`OPENROUTER_API_BASE`** | Optional override |
| **`HSM_AGENT_CHAT_MODEL`** / **`DEFAULT_LLM_MODEL`** | Chat model id (e.g. Opus-class routing via OpenRouter id) |
| **`HSM_AGENT_CHAT_PROVIDER`** | `openrouter` vs `ollama` |

## Meta-harness worker routing

| Env | Default | Role |
|-----|---------|------|
| **`HSM_META_HARNESS_FORCE_WORKER`** | `1` (on) | Adds **`forceWorkerDispatch: true`** in **`evaluate_turn.py`** so streams include **runtime** tool telemetry |

Set to **`0`** only when intentionally testing **OpenRouter-only** conversational turns (different score shape).

## Optional production temporal path (Console only)

These apply to **`web/company-console`** conversational LLM — **not** `services/claude-harness`:

| Env | Role |
|-----|------|
| **`HSM_AGENT_CHAT_TEMPORAL_TWO_PASS`** | `1` = timeline extract → answer for temporal-looking questions |
| **`HSM_AGENT_CHAT_TEMPORAL_VERIFY`** | `1` = third pass SUPPORTED/UNSUPPORTED vs thread corpus |
| **`HSM_AGENT_CHAT_TEMPORAL_MIN_CORPUS_CHARS`** | Minimum thread size before two-pass (default `280`) |

## Claude harness vs agent-chat (explicit split)

| Surface | Role |
|---------|------|
| **`services/claude-harness`** | Runs **Claude Code** CLI → **`CompletionEvent`** NDJSON; session files under **`HSM_CLAUDE_HARNESS_SESSION_DIR`**. **Does not** implement Console temporal two-pass. |
| **`POST /api/agent-chat-reply/stream`** | **Operator chat** — worker + OpenRouter/Ollama + optional temporal passes. This is what **meta-harness** drives for Company OS scoring. |
| **`services/agent-chat-harness`** | Thin **HTTP driver** on port **3849** → proxies **`/api/agent-chat-reply/stream`** and logs sessions. **Automation only**; quality = Next + Rust + model. |

**LongMemEval / temporal reasoning benchmarks** line up with **OpenRouter + task memory + prompts** (and Rust **`hsm-longmemeval`**). **Claude harness** line up with **tool telemetry + execution shape** for meta-harness-style gates when execution backend is Claude — not the same code path as Console temporal env flags.

## Regression smoke thresholds

Script: **`scripts/company-os-agent-chat-meta-harness-smoke.sh`**

| Env | Default | Role |
|-----|---------|------|
| **`MH_MIN_MEAN_SCORE`** | `0.6` | Mean composite score gate |
| **`MH_MIN_FINALIZE_RATE`** | `0.65` | Fraction of turns finalized — tolerate occasional flaky worker telemetry locally; raise toward **`1.0`** for strict CI |

## Suggested model matrix (document yours here)

Record what you actually run so “good score” is reproducible:

| Slot | Example | Notes |
|------|---------|--------|
| **Primary regression** | Your OpenRouter id for Opus-class | Paste the exact **`HSM_AGENT_CHAT_MODEL`** string |
| **Fast smoke** | Cheap/free OpenRouter model | For PRs |
| **Local** | Ollama model | When **`HSM_AGENT_CHAT_PROVIDER=ollama`** |

## One-liner checks

```bash
# Meta-harness paper smoke (Next + hsm_console + Python)
bash scripts/company-os-agent-chat-meta-harness-smoke.sh

# Endpoint gate (reply + stream)
bash scripts/agent-chat-endpoint-check.sh
```

## Related docs

- **`docs/EVAL_AND_META_HARNESS.md`** — meta-harness vs eval binaries
- **`services/claude-harness/README.md`** — Claude Code harness
- **`services/agent-chat-harness/README.md`** — optional NDJSON driver