Company OS Continual Harness implementation plan

This is the implementation roadmap for bringing HSM-II Company OS closer to Continual Harness (paper PDF) while preserving the Company OS safety contract: observe first, write artifacts, review deltas, re-test, then promote.

1. Current state

Company OS now has the first safe slice:

meta_harness.py / evaluate_turn.py
  -> results.json + results/*.ndjson
  -> company_os_refiner_pass.py
  -> refiner/company_os_turns.jsonl
  -> refiner/company_os_harness_delta_<n>.json

Implemented files:

• scripts/meta-harness/evaluate_turn.py — live Next + hsm_console turn observation and scoring.
• scripts/meta-harness/meta_harness.py — candidate evaluation and frontier logging.
• scripts/meta-harness/company_os_refiner_pass.py — normalizes live agent-chat observations and emits review-only deltas.
• scripts/meta-harness/test_company_os_refiner_pass.py — tests the trace-to-delta path.
• scripts/meta-harness/harness_delta_filled.py — shared validation for filled deltas and applier.
• scripts/meta-harness/test_company_os_llm_refiner.py / test_company_os_delta_apply.py — M1/M2 smoke tests.
• scripts/meta-harness/company_os_llm_refiner.py — filled delta generation (mock or OpenRouter).
• scripts/meta-harness/company_os_delta_apply.py — versioned hsm.company_os_harness_policy.v1 JSON.
• scripts/meta-harness/company_os_promote_delta.py — promotion dry-run gate.

This gives Company OS observe -> artifact -> detect -> structured candidate. An LLM Refiner (company_os_llm_refiner.py) and policy applier (company_os_delta_apply.py + optional HSM_COMPANY_OS_HARNESS_POLICY_PATH) are in-tree; operators still gate activation and re-test. It does not yet run task-local online overlays (M4) or model training (M5).

2. Target architecture

Map the paper's H = (p, G, K, M) to Company OS surfaces:

Paper component	Company OS target	Apply mode
p prompt	web/company-console/app/lib/agent-chat-prompt-policy.mjs, worker companion prompt, operator ethos copy	Feature-flagged patch or policy JSON
G sub-agents	company_agents, persona/agent templates, task spawn policy	Postgres row changes or reviewed bundle import
K skills/tools	capability_refs, skill detector rules, tool allowlists, skills catalog	Reviewed JSON patch + task-scoped allowlist
M memory	company_memory_entries, llm-context retrieval policy, broadcast/shared/agent scopes	Memory rows + retrieval config
trajectory τ	company_os_turns.jsonl, NDJSON stream logs, agent runs, governance events	Append-only artifacts
delta Δ	company_os_harness_delta_<n>.json	Review-only until promoted

The invariant: no refiner output becomes runtime behavior until it has a provenance path, a bounded target, a re-test, and a rollback.

3. Milestones

M1 — LLM Refiner that fills deltas

Goal: turn deterministic placeholder deltas into concrete but still review-only proposals.

Implementation:

• scripts/meta-harness/company_os_llm_refiner.py — mock-by-default; OpenRouter when OPENROUTER_API_KEY is set.
• Input: company_os_turns.jsonl, company_os_harness_delta_<n>.json, relevant code/doc excerpts.
• Output: company_os_harness_delta_<n>.filled.json.
• Use a strict JSON schema with these top-level sections:
  • prompt_policy_patch
  • skill_dispatch_patch
  • runtime_guard_patch
  • memory_policy_patch
  • sub_agent_patch
  • risk_notes
  • verification_plan
• Require apply_policy: review_only.
• Reject any patch that targets a file/path/API outside the allowlist.

Verification:

python3 scripts/meta-harness/test_company_os_refiner_pass.py
python3 scripts/meta-harness/test_company_os_llm_refiner.py
PYTHONPYCACHEPREFIX=/private/tmp/hsm-refiner-pyc python3 -m py_compile scripts/meta-harness/*.py

Acceptance:

• Given a weak-turn artifact, the LLM Refiner produces valid JSON.
• Invalid target paths are rejected.
• Missing verification plan fails validation.

M2 — Approval-gated applier

Goal: safely map reviewed deltas into runtime behavior behind flags.

Implementation:

• scripts/meta-harness/company_os_delta_apply.py — versioned policy JSON + optional --activate.
• Supported apply targets only:
  • policy JSON under a new config/company_os_harness/ directory,
  • generated review patch files under candidate directories,
  • optional Postgres memory/task suggestions exported as JSON, not directly written by default.
• Do not edit TypeScript/Rust source directly in the first applier.
• Add HSM_COMPANY_OS_HARNESS_POLICY_PATH consumed by agent-chat server code after review.
• Add rollback by preserving prior policy JSON and candidate metadata.

Verification:

python3 scripts/meta-harness/test_company_os_delta_apply.py
npm --prefix web/company-console run test -- --runInBand
npm --prefix docs-site run build

Acceptance:

• Applying a valid policy delta creates a versioned policy file.
• Reverting restores the previous policy.
• Agent-chat reads the policy only when explicitly configured.

M3 — Re-test and promotion loop

Goal: close the safe loop: delta -> apply candidate policy -> re-run meta-harness -> promote if improved.

Implementation:

• scripts/meta-harness/company_os_promote_delta.py — --dry-run metrics gate (full promote TBD).
• Promotion requires:
  • source candidate ID,
  • filled delta,
  • applied policy path,
  • before/after metrics,
  • no regression on finalize_rate,
  • explicit operator approval or --approved-by.
• Write promotion record to candidate dir and optionally Company OS governance event when API credentials are present.

Verification:

bash scripts/company-os-agent-chat-meta-harness-smoke.sh
python3 scripts/meta-harness/company_os_promote_delta.py --dry-run ...
python3 scripts/meta-harness/test_company_os_promote_and_m5.py

Acceptance:

• Promotion refuses deltas without after-metrics.
• Promotion refuses lower finalize_rate unless --allow-regression is explicit.

M4 — Online in-run refinement

Goal: approximate the paper's H <- H ⊕ Δ during long agent-chat/operator sessions, still with gates.

Implementation:

• POST /api/company/tasks/{task_id}/harness-overlay — apply { "apply": { "text": "..." } } or clear { "clear": true }; logged to governance_events.

• scripts/meta-harness/company_os_task_overlay.py — operator CLI wrapping the same API.

• src/company_os/agents.rs — llm-context renders Task-local harness overlay (reversible) separately from stigmergic handoff notes.

• Keep refinements task-scoped first.

• Allow runtime to load a task-local policy overlay between turns, not mid-token.

• Record overlay IDs in task context_notes and run metadata.

• Expose active overlay in llm-context.

Safety:

• Only reversible overlays.
• No source-file mutation.
• Human approval required for shared/company-wide overlays.

Acceptance:

• A long-running task can adopt a task-local overlay after a weak turn.
• The overlay appears in llm-context.
• Removing the overlay restores baseline behavior.

CLI (shipped): scripts/meta-harness/company_os_overlay_from_delta.py applies a validated *.filled.json to POST .../harness-overlay. Meta-harness: --apply-overlay-task-id <uuid> (or HSM_COMPANY_OS_OVERLAY_TASK_ID).

M5 — PRM / teacher relabel / SFT lane

Goal: separate training lane for model co-learning, not part of live production control.

Implementation:

• schemas/company_os_rollout_shard.v1.json — shard contract.

• scripts/meta-harness/company_os_training_export.py — JSONL turns → shard files under runs/company_os_training/.

• scripts/meta-harness/company_os_prm_score.py — heuristic PRM scores per window row.

• scripts/meta-harness/company_os_training_dry_run.py — token/label stats without training.

• Define rollout shard schema under schemas/company_os_rollout_shard.v1.json.

• Export company_os_turns.jsonl windows into teacher-label jobs.

• Add PRM scoring interface:

  • start with heuristic/process scores from evaluate_turn.py,
  • later allow frontier teacher labels.

• Store training data under runs/company_os_training/.

• Add a dry-run trainer command before real SFT:

  • validates data,
  • computes label distributions,
  • estimates token/compute cost.

Acceptance:

• Can export shards without training.
• Can score/relabel shards without touching production.
• Any SFT adapter requires a held-out Company OS meta-harness eval before use.

4. Safety gates

Every stage must preserve these gates:

• Allowlist gate: refiner/applier can only target known policy surfaces.
• Schema gate: all deltas validate before apply.
• Review gate: runtime changes are not automatic.
• Re-test gate: candidate policy must be evaluated after apply.
• Rollback gate: every promoted policy has a previous version.
• Ledger gate: promotions are recorded in artifacts and, when online, Company OS governance events.

5. Near-term implementation order

1. Implement company_os_llm_refiner.py with schema validation and tests.
2. Add policy file target under config/company_os_harness/.
3. Implement company_os_delta_apply.py in dry-run/versioned-policy mode.
4. Teach agent-chat to optionally read HSM_COMPANY_OS_HARNESS_POLICY_PATH.
5. Add promotion command and smoke workflow.
6. Only then consider task-local online overlays.