Research and earlier design

Last changed 2026-05-22. This page records research, planning, or an earlier architecture and may not describe the current product. For current behaviour, start at Start here.

Company OS Continual Harness implementation plan

This is the implementation roadmap for bringing HSM-II Company OS closer to Continual Harness (paper PDF) while preserving the Company OS safety contract: observe first, write artifacts, review deltas, re-test, then promote.

1. Current state

Company OS now has the first safe slice:

meta_harness.py / evaluate_turn.py
  -> results.json + results/*.ndjson
  -> company_os_refiner_pass.py
  -> refiner/company_os_turns.jsonl
  -> refiner/company_os_harness_delta_<n>.json

Implemented files:

• scripts/meta-harness/evaluate_turn.py — live Next + hsm_console turn observation and scoring.
• scripts/meta-harness/meta_harness.py — candidate evaluation and frontier logging.
• scripts/meta-harness/company_os_refiner_pass.py — normalizes live agent-chat observations and emits review-only deltas.
• scripts/meta-harness/test_company_os_refiner_pass.py — tests the trace-to-delta path.
• scripts/meta-harness/harness_delta_filled.py — shared validation for filled deltas and applier.
• scripts/meta-harness/test_company_os_llm_refiner.py / test_company_os_delta_apply.py — M1/M2 smoke tests.
• scripts/meta-harness/company_os_llm_refiner.py — filled delta generation (mock or OpenRouter).
• scripts/meta-harness/company_os_delta_apply.py — versioned hsm.company_os_harness_policy.v1 JSON.
• scripts/meta-harness/company_os_promote_delta.py — promotion dry-run gate.

This gives Company OS observe -> artifact -> detect -> structured candidate. An LLM Refiner (company_os_llm_refiner.py) and policy applier (company_os_delta_apply.py + optional HSM_COMPANY_OS_HARNESS_POLICY_PATH) are in-tree; operators still gate activation and re-test. It does not yet run task-local online overlays (M4) or model training (M5).

2. Target architecture

Map the paper's H = (p, G, K, M) to Company OS surfaces:

Paper component	Company OS target	Apply mode
p prompt	web/company-console/app/lib/agent-chat-prompt-policy.mjs, worker companion prompt, operator ethos copy	Feature-flagged patch or policy JSON
G sub-agents	company_agents, persona/agent templates, task spawn policy	Postgres row changes or reviewed bundle import
K skills/tools	capability_refs, skill detector rules, tool allowlists, skills catalog	Reviewed JSON patch + task-scoped allowlist
M memory	company_memory_entries, llm-context retrieval policy, broadcast/shared/agent scopes	Memory rows + retrieval config
trajectory τ	company_os_turns.jsonl, NDJSON stream logs, agent runs, governance events	Append-only artifacts
delta Δ	company_os_harness_delta_<n>.json	Review-only until promoted

The invariant: no refiner output becomes runtime behavior until it has a provenance path, a bounded target, a re-test, and a rollback.

3. Milestones

M1 — LLM Refiner that fills deltas

Goal: turn deterministic placeholder deltas into concrete but still review-only proposals.

Implementation:

• scripts/meta-harness/company_os_llm_refiner.py — mock-by-default; OpenRouter when OPENROUTER_API_KEY is set.
• Input: company_os_turns.jsonl, company_os_harness_delta_<n>.json, relevant code/doc excerpts.
• Output: company_os_harness_delta_<n>.filled.json.
• Use a strict JSON schema with these top-level sections:
  • prompt_policy_patch
  • skill_dispatch_patch
  • runtime_guard_patch
  • memory_policy_patch
  • sub_agent_patch
  • risk_notes
  • verification_plan
• Require apply_policy: review_only.
• Reject any patch that targets a file/path/API outside the allowlist.

Verification:

python3 scripts/meta-harness/test_company_os_refiner_pass.py
python3 scripts/meta-harness/test_company_os_llm_refiner.py
PYTHONPYCACHEPREFIX=/private/tmp/hsm-refiner-pyc python3 -m py_compile scripts/meta-harness/*.py

Acceptance:

• Given a weak-turn artifact, the LLM Refiner produces valid JSON.
• Invalid target paths are rejected.
• Missing verification plan fails validation.

M2 — Approval-gated applier

Goal: safely map reviewed deltas into runtime behavior behind flags.

Implementation:

• scripts/meta-harness/company_os_delta_apply.py — versioned policy JSON + optional --activate.
• Supported apply targets only:
  • policy JSON under a new config/company_os_harness/ directory,
  • generated review patch files under candidate directories,
  • optional Postgres memory/task suggestions exported as JSON, not directly written by default.
• Do not edit TypeScript/Rust source directly in the first applier.
• Add HSM_COMPANY_OS_HARNESS_POLICY_PATH consumed by agent-chat server code after review.
• Add rollback by preserving prior policy JSON and candidate metadata.

Verification:

python3 scripts/meta-harness/test_company_os_delta_apply.py
npm --prefix web/company-console run test -- --runInBand
npm --prefix docs-site run build

Acceptance:

• Applying a valid policy delta creates a versioned policy file.
• Reverting restores the previous policy.
• Agent-chat reads the policy only when explicitly configured.

M3 — Re-test and promotion loop

Goal: close the safe loop: delta -> apply candidate policy -> re-run meta-harness -> promote if improved.

Implementation:

• scripts/meta-harness/company_os_promote_delta.py — --dry-run metrics gate (full promote TBD).
• Promotion requires:
  • source candidate ID,
  • filled delta,
  • applied policy path,
  • before/after metrics,
  • no regression on finalize_rate,
  • explicit operator approval or --approved-by.
• Write promotion record to candidate dir and optionally Company OS governance event when API credentials are present.

Verification:

bash scripts/company-os-agent-chat-meta-harness-smoke.sh
python3 scripts/meta-harness/company_os_promote_delta.py --dry-run ...
python3 scripts/meta-harness/test_company_os_promote_and_m5.py

Acceptance:

• Promotion refuses deltas without after-metrics.
• Promotion refuses lower finalize_rate unless --allow-regression is explicit.

M4 — Online in-run refinement

Goal: approximate the paper's H <- H ⊕ Δ during long agent-chat/operator sessions, still with gates.

Implementation:

• POST /api/company/tasks/{task_id}/harness-overlay — apply { "apply": { "text": "..." } } or clear { "clear": true }; logged to governance_events.

• scripts/meta-harness/company_os_task_overlay.py — operator CLI wrapping the same API.

• src/company_os/agents.rs — llm-context renders Task-local harness overlay (reversible) separately from stigmergic handoff notes.

• Keep refinements task-scoped first.

• Allow runtime to load a task-local policy overlay between turns, not mid-token.

• Record overlay IDs in task context_notes and run metadata.

• Expose active overlay in llm-context.

Safety:

• Only reversible overlays.
• No source-file mutation.
• Human approval required for shared/company-wide overlays.

Acceptance:

• A long-running task can adopt a task-local overlay after a weak turn.
• The overlay appears in llm-context.
• Removing the overlay restores baseline behavior.

CLI (shipped): scripts/meta-harness/company_os_overlay_from_delta.py applies a validated *.filled.json to POST .../harness-overlay. Meta-harness: --apply-overlay-task-id <uuid> (or HSM_COMPANY_OS_OVERLAY_TASK_ID).

M5 — PRM / teacher relabel / SFT lane

Goal: separate training lane for model co-learning, not part of live production control.

Implementation:

• schemas/company_os_rollout_shard.v1.json — shard contract.

• scripts/meta-harness/company_os_training_export.py — JSONL turns → shard files under runs/company_os_training/.

• scripts/meta-harness/company_os_prm_score.py — heuristic PRM scores per window row.

• scripts/meta-harness/company_os_training_dry_run.py — token/label stats without training.

• Define rollout shard schema under schemas/company_os_rollout_shard.v1.json.

• Export company_os_turns.jsonl windows into teacher-label jobs.

• Add PRM scoring interface:

  • start with heuristic/process scores from evaluate_turn.py,
  • later allow frontier teacher labels.

• Store training data under runs/company_os_training/.

• Add a dry-run trainer command before real SFT:

  • validates data,
  • computes label distributions,
  • estimates token/compute cost.

Acceptance:

• Can export shards without training.
• Can score/relabel shards without touching production.
• Any SFT adapter requires a held-out Company OS meta-harness eval before use.

4. Safety gates

Every stage must preserve these gates:

• Allowlist gate: refiner/applier can only target known policy surfaces.
• Schema gate: all deltas validate before apply.
• Review gate: runtime changes are not automatic.
• Re-test gate: candidate policy must be evaluated after apply.
• Rollback gate: every promoted policy has a previous version.
• Ledger gate: promotions are recorded in artifacts and, when online, Company OS governance events.

5. Near-term implementation order

1. Implement company_os_llm_refiner.py with schema validation and tests.
2. Add policy file target under config/company_os_harness/.
3. Implement company_os_delta_apply.py in dry-run/versioned-policy mode.
4. Teach agent-chat to optionally read HSM_COMPANY_OS_HARNESS_POLICY_PATH.
5. Add promotion command and smoke workflow.
6. Only then consider task-local online overlays.