Research and earlier design

Last changed 2026-04-07. This page records research, planning, or an earlier architecture and may not describe the current product. For current behaviour, start at Start here.

Benchmark Stack

This repo should carry three benchmark tracks:

1. LongMemEval for memory fidelity
2. YC-Bench for long-horizon outcomes
3. HSM-native for stigmergic memory, belief revision, and agent handoff

The goal is not to force one benchmark to prove everything. Each track should measure one thing well.

1. LongMemEval

Purpose:

• Verify that HSM-II preserves exact cross-session memory without lossy abstraction.

Current entrypoint:

• src/bin/hsm_longmemeval.rs

Files to add or maintain:

• src/bin/hsm_longmemeval.rs
• src/eval/runner.rs
• src/eval/trace.rs
• src/eval/metrics.rs
• scripts/run_longmemeval.sh
• scripts/eval_longmemeval.sh
• docs/LONGMEMEVAL.md

Required modes:

• baseline-direct Raw timestamped history, no HSM retrieval layer.
• hsm-fullhistory-plus-memory Raw timestamped history plus HSM memory augmentation.
• hsm-retrieval-only Retrieved evidence without full history. This is an ablation, not the default benchmark mode.

Required commands:

cargo run --bin hsm-longmemeval -- \
  --input external/LongMemEval/data/longmemeval_oracle.json \
  --output runs/longmemeval/baseline_oracle.jsonl \
  --mode baseline

cargo run --bin hsm-longmemeval -- \
  --input external/LongMemEval/data/longmemeval_oracle.json \
  --output runs/longmemeval/hsm_oracle.jsonl \
  --mode hsm

Required output files:

• *.jsonl predictions with:

{"question_id":"...", "hypothesis":"..."}

• *.trace.jsonl with retrieval/debug metadata
• *.eval.json aggregate summary

Required summary schema:

{
  "benchmark": "longmemeval",
  "mode": "baseline-direct",
  "dataset": "oracle",
  "model": "openai/gpt-5.4",
  "n_questions": 500,
  "qa_accuracy": 0.0,
  "abstention_accuracy": 0.0,
  "retrieval_turn_recall_at_k": 0.0,
  "retrieval_session_recall_at_k": 0.0,
  "notes": []
}

Success criteria:

• hsm-fullhistory-plus-memory should stay near baseline-direct on oracle mode.
• hsm-retrieval-only is allowed to underperform, but should improve as retrieval quality improves.

2. YC-Bench

Purpose:

• Measure whether memory improves long-horizon company performance under compounding consequences.

Current integration points:

• external_integrations/auto-harness-yc-bench/
• config/external_yc_bench_seed*.json
• web/company-console/app/api/companies-sh/yc-bench/

Files to add or maintain:

• scripts/run_ycbench_grid.sh
• scripts/aggregate_ycbench_hsm.py
• scripts/compare_ycbench_ablations.py
• docs/YC_BENCH.md
• config/ycbench_hsm_baseline.json
• config/ycbench_hsm_nomemory.json
• config/ycbench_hsm_scratchpad_only.json
• config/ycbench_hsm_full.json

Required ablations:

• baseline-no-memory
• scratchpad-only
• hsm-memory-no-belief-update
• hsm-full

Required outputs per run:

{
  "benchmark": "yc-bench",
  "company": "apex-systems",
  "seed": 1,
  "model": "openrouter/qwen/qwen3.6-plus:free",
  "variant": "hsm-full",
  "final_funds_ratio": 0.0,
  "terminal_reason": "horizon_end",
  "turns_completed": 0,
  "total_cost_usd": 0.0
}

Required aggregate schema:

{
  "benchmark": "yc-bench",
  "variant": "hsm-full",
  "val_score": 0.0,
  "pass_rate": 0.0,
  "pass_count": 0,
  "n_companies": 0,
  "avg_seeds": 0.0,
  "companies": {},
  "tier_distribution": {},
  "behavioral_metrics": {
    "rat_accept_rate": 0.0,
    "trust_task_ratio": 0.0,
    "policy_flip_rate": 0.0,
    "bankruptcy_rate": 0.0
  }
}

Success criteria:

• hsm-full should beat weaker-memory variants on funds ratio and behavioral stability.
• Behavioral metrics should explain the outcome, not just the final dollars.

3. HSM-Native

Purpose:

• Measure what HSM-II is actually designed for: stigmergic coordination, belief revision, agent handoff, and cross-session synthesis.

New files to add:

• src/eval/hsm_native_tasks.rs
• src/bin/hsm_native_eval.rs
• src/eval/hsm_native_metrics.rs
• docs/HSM_NATIVE_BENCH.md
• scripts/run_hsm_native.sh

Suites to add:

• cross_session_synthesis
• belief_revision
• agent_handoff
• policy_persistence
• conflict_resolution

Task schema:

{
  "id": "handoff-001",
  "suite": "agent_handoff",
  "sessions": [
    {
      "session_id": 1,
      "agent": "researcher",
      "turns": []
    }
  ],
  "question": "What should the finisher do next?",
  "gold": {
    "answer": "...",
    "required_facts": [],
    "forbidden_stale_facts": []
  }
}

Required metrics schema:

{
  "benchmark": "hsm-native",
  "suite": "belief_revision",
  "variant": "hsm-full",
  "n_tasks": 0,
  "answer_accuracy": 0.0,
  "required_fact_recall": 0.0,
  "stale_fact_suppression": 0.0,
  "handoff_success": 0.0,
  "policy_consistency": 0.0,
  "explanation_grounding": 0.0
}

Success criteria:

• HSM should clearly outperform plain chat-history baselines on belief revision and handoff tasks.

Implementation Order

1. Stabilize LongMemEval as a regression suite.
2. Add YC-Bench ablation variants and behavioral aggregates.
3. Build HSM-native benchmark once the first two tracks are reproducible.

Short Task List

LongMemEval

• Add direct/baseline/hsm mode naming cleanup in src/bin/hsm_longmemeval.rs
• Export retrieval metrics from traces
• Add scripts/run_longmemeval.sh
• Add scripts/eval_longmemeval.sh
• Add JSON summary writer

YC-Bench

• Add named ablation configs under config/
• Add run matrix script under scripts/
• Add behavior aggregator for RAT avoidance, trust specialization, policy stability
• Store aggregate outputs under runs/ycbench/

HSM-Native

• Add task format under src/eval/
• Add runner bin under src/bin/
• Add scorers under src/eval/
• Add seedable JSONL artifact output

Repository Conventions

Suggested run artifact layout:

runs/
  longmemeval/
  ycbench/
  hsm_native/

Suggested docs:

docs/
  LONGMEMEVAL.md
  YC_BENCH.md
  HSM_NATIVE_BENCH.md
  BENCHMARK_STACK.md