Reference harness profile (agent-chat + meta-harness + LongMemEval)
This is the single stack to use when you want comparable regression numbers across machines and models. Change these env vars and you are not doing an apples-to-apples run.
Two gates (do not conflate them)
| Gate | What it measures | Primary commands / surfaces |
|---|---|---|
| Meta-harness | Worker stream health — NDJSON from POST /api/agent-chat-reply/stream: phases, runtime / runtime_raw (CompletionEvent-shaped tool telemetry), done with reply + finalized, errors, composite score | scripts/company-os-agent-chat-meta-harness-smoke.sh, scripts/meta-harness/evaluate_turn.py, scripts/meta-harness/meta_harness.py |
| LongMemEval | Memory QA accuracy — oracle match / temporal reasoning vs haystack JSON | cargo run --bin hsm-longmemeval …, cargo run --bin hsm-longmemeval-agent-chat … (Rust API parity flags); optional hsm-eval / HsmRunner tuning |
High meta-harness scores usually require forceWorkerDispatch so turns hit operator-chat → execute-worker (tool counts + runtime lines). LongMemEval cares about answers, not NDJSON tool density.
Locked URLs (defaults)
| Env | Default | Meaning |
|---|---|---|
NEXT_URL | http://127.0.0.1:3050 | Company Console (Next) — meta-harness reads this |
HSM_URL / HSM_CONSOLE_URL | http://127.0.0.1:3847 | Company OS (hsm_console) |
HSM_COMPANY_CONSOLE_URL | same idea | Next origin for services/agent-chat-harness HTTP driver |
Meta-harness Python defaults align with evaluate_turn.py (NEXT_URL, HSM_URL).
Operator chat / LLM (Next server-side)
Set these on the Company Console process (the Next app), not only in the shell running smoke tests:
| Env | Role |
|---|---|
OPENROUTER_API_KEY | Conversational path + many smoke configs |
HSM_OPENROUTER_API_KEY | Alternate name picked up by Next |
OPENROUTER_API_BASE | Optional override |
HSM_AGENT_CHAT_MODEL / DEFAULT_LLM_MODEL | Chat model id (e.g. Opus-class routing via OpenRouter id) |
HSM_AGENT_CHAT_PROVIDER | openrouter vs ollama |
Meta-harness worker routing
| Env | Default | Role |
|---|---|---|
HSM_META_HARNESS_FORCE_WORKER | 1 (on) | Adds forceWorkerDispatch: true in evaluate_turn.py so streams include runtime tool telemetry |
Set to 0 only when intentionally testing OpenRouter-only conversational turns (different score shape).
Optional production temporal path (Console only)
These apply to web/company-console conversational LLM — not services/claude-harness:
| Env | Role |
|---|---|
HSM_AGENT_CHAT_TEMPORAL_TWO_PASS | 1 = timeline extract → answer for temporal-looking questions |
HSM_AGENT_CHAT_TEMPORAL_VERIFY | 1 = third pass SUPPORTED/UNSUPPORTED vs thread corpus |
HSM_AGENT_CHAT_TEMPORAL_MIN_CORPUS_CHARS | Minimum thread size before two-pass (default 280) |
Claude harness vs agent-chat (explicit split)
| Surface | Role |
|---|---|
services/claude-harness | Runs Claude Code CLI → CompletionEvent NDJSON; session files under HSM_CLAUDE_HARNESS_SESSION_DIR. Does not implement Console temporal two-pass. |
POST /api/agent-chat-reply/stream | Operator chat — worker + OpenRouter/Ollama + optional temporal passes. This is what meta-harness drives for Company OS scoring. |
services/agent-chat-harness | Thin HTTP driver on port 3849 → proxies /api/agent-chat-reply/stream and logs sessions. Automation only; quality = Next + Rust + model. |
LongMemEval / temporal reasoning benchmarks line up with OpenRouter + task memory + prompts (and Rust hsm-longmemeval). Claude harness line up with tool telemetry + execution shape for meta-harness-style gates when execution backend is Claude — not the same code path as Console temporal env flags.
Regression smoke thresholds
Script: scripts/company-os-agent-chat-meta-harness-smoke.sh
| Env | Default | Role |
|---|---|---|
MH_MIN_MEAN_SCORE | 0.6 | Mean composite score gate |
MH_MIN_FINALIZE_RATE | 0.65 | Fraction of turns finalized — tolerate occasional flaky worker telemetry locally; raise toward 1.0 for strict CI |
Suggested model matrix (document yours here)
Record what you actually run so “good score” is reproducible:
| Slot | Example | Notes |
|---|---|---|
| Primary regression | Your OpenRouter id for Opus-class | Paste the exact HSM_AGENT_CHAT_MODEL string |
| Fast smoke | Cheap/free OpenRouter model | For PRs |
| Local | Ollama model | When HSM_AGENT_CHAT_PROVIDER=ollama |
One-liner checks
bash
# Meta-harness paper smoke (Next + hsm_console + Python)
bash scripts/company-os-agent-chat-meta-harness-smoke.sh
# Endpoint gate (reply + stream)
bash scripts/agent-chat-endpoint-check.shRelated docs
docs/EVAL_AND_META_HARNESS.md— meta-harness vs eval binariesservices/claude-harness/README.md— Claude Code harnessservices/agent-chat-harness/README.md— optional NDJSON driver