# Company OS Continual Harness implementation plan

This is the implementation roadmap for bringing HSM-II Company OS closer to **Continual Harness** ([paper PDF](https://arxiv.org/pdf/2605.09998)) while preserving the Company OS safety contract: observe first, write artifacts, review deltas, re-test, then promote.

## 1. Current state

Company OS now has the first safe slice:

```text
meta_harness.py / evaluate_turn.py
  -> results.json + results/*.ndjson
  -> company_os_refiner_pass.py
  -> refiner/company_os_turns.jsonl
  -> refiner/company_os_harness_delta_<n>.json
```

Implemented files:

- `scripts/meta-harness/evaluate_turn.py` — live Next + `hsm_console` turn observation and scoring.
- `scripts/meta-harness/meta_harness.py` — candidate evaluation and frontier logging.
- `scripts/meta-harness/company_os_refiner_pass.py` — normalizes live agent-chat observations and emits review-only deltas.
- `scripts/meta-harness/test_company_os_refiner_pass.py` — tests the trace-to-delta path.
- `scripts/meta-harness/harness_delta_filled.py` — shared validation for filled deltas and applier.
- `scripts/meta-harness/test_company_os_llm_refiner.py` / `test_company_os_delta_apply.py` — M1/M2 smoke tests.
- `scripts/meta-harness/company_os_llm_refiner.py` — filled delta generation (mock or OpenRouter).
- `scripts/meta-harness/company_os_delta_apply.py` — versioned `hsm.company_os_harness_policy.v1` JSON.
- `scripts/meta-harness/company_os_promote_delta.py` — promotion dry-run gate.

This gives Company OS **observe -> artifact -> detect -> structured candidate**. An **LLM Refiner** (`company_os_llm_refiner.py`) and **policy applier** (`company_os_delta_apply.py` + optional `HSM_COMPANY_OS_HARNESS_POLICY_PATH`) are in-tree; operators still gate activation and re-test. It does **not** yet run task-local online overlays (M4) or model training (M5).

## 2. Target architecture

Map the paper's `H = (p, G, K, M)` to Company OS surfaces:

| Paper component | Company OS target | Apply mode |
|-----------------|-------------------|------------|
| `p` prompt | `web/company-console/app/lib/agent-chat-prompt-policy.mjs`, worker companion prompt, operator ethos copy | Feature-flagged patch or policy JSON |
| `G` sub-agents | `company_agents`, persona/agent templates, task spawn policy | Postgres row changes or reviewed bundle import |
| `K` skills/tools | `capability_refs`, skill detector rules, tool allowlists, skills catalog | Reviewed JSON patch + task-scoped allowlist |
| `M` memory | `company_memory_entries`, `llm-context` retrieval policy, broadcast/shared/agent scopes | Memory rows + retrieval config |
| trajectory `τ` | `company_os_turns.jsonl`, NDJSON stream logs, agent runs, governance events | Append-only artifacts |
| delta `Δ` | `company_os_harness_delta_<n>.json` | Review-only until promoted |

The invariant: no refiner output becomes runtime behavior until it has a provenance path, a bounded target, a re-test, and a rollback.

## 3. Milestones

### M1 — LLM Refiner that fills deltas

Goal: turn deterministic placeholder deltas into concrete but still review-only proposals.

Implementation:

- `scripts/meta-harness/company_os_llm_refiner.py` — mock-by-default; OpenRouter when `OPENROUTER_API_KEY` is set.
- Input: `company_os_turns.jsonl`, `company_os_harness_delta_<n>.json`, relevant code/doc excerpts.
- Output: `company_os_harness_delta_<n>.filled.json`.
- Use a strict JSON schema with these top-level sections:
  - `prompt_policy_patch`
  - `skill_dispatch_patch`
  - `runtime_guard_patch`
  - `memory_policy_patch`
  - `sub_agent_patch`
  - `risk_notes`
  - `verification_plan`
- Require `apply_policy: review_only`.
- Reject any patch that targets a file/path/API outside the allowlist.

Verification:

```bash
python3 scripts/meta-harness/test_company_os_refiner_pass.py
python3 scripts/meta-harness/test_company_os_llm_refiner.py
PYTHONPYCACHEPREFIX=/private/tmp/hsm-refiner-pyc python3 -m py_compile scripts/meta-harness/*.py
```

Acceptance:

- Given a weak-turn artifact, the LLM Refiner produces valid JSON.
- Invalid target paths are rejected.
- Missing verification plan fails validation.

### M2 — Approval-gated applier

Goal: safely map reviewed deltas into runtime behavior behind flags.

Implementation:

- `scripts/meta-harness/company_os_delta_apply.py` — versioned policy JSON + optional `--activate`.
- Supported apply targets only:
  - policy JSON under a new `config/company_os_harness/` directory,
  - generated review patch files under candidate directories,
  - optional Postgres memory/task suggestions exported as JSON, not directly written by default.
- Do **not** edit TypeScript/Rust source directly in the first applier.
- Add `HSM_COMPANY_OS_HARNESS_POLICY_PATH` consumed by agent-chat server code after review.
- Add rollback by preserving prior policy JSON and candidate metadata.

Verification:

```bash
python3 scripts/meta-harness/test_company_os_delta_apply.py
npm --prefix web/company-console run test -- --runInBand
npm --prefix docs-site run build
```

Acceptance:

- Applying a valid policy delta creates a versioned policy file.
- Reverting restores the previous policy.
- Agent-chat reads the policy only when explicitly configured.

### M3 — Re-test and promotion loop

Goal: close the safe loop: delta -> apply candidate policy -> re-run meta-harness -> promote if improved.

Implementation:

- `scripts/meta-harness/company_os_promote_delta.py` — `--dry-run` metrics gate (full promote TBD).
- Promotion requires:
  - source candidate ID,
  - filled delta,
  - applied policy path,
  - before/after metrics,
  - no regression on `finalize_rate`,
  - explicit operator approval or `--approved-by`.
- Write promotion record to candidate dir and optionally Company OS governance event when API credentials are present.

Verification:

```bash
bash scripts/company-os-agent-chat-meta-harness-smoke.sh
python3 scripts/meta-harness/company_os_promote_delta.py --dry-run ...
python3 scripts/meta-harness/test_company_os_promote_and_m5.py
```

Acceptance:

- Promotion refuses deltas without after-metrics.
- Promotion refuses lower `finalize_rate` unless `--allow-regression` is explicit.

### M4 — Online in-run refinement

Goal: approximate the paper's `H <- H ⊕ Δ` during long agent-chat/operator sessions, still with gates.

Implementation:
- `POST /api/company/tasks/{task_id}/harness-overlay` — apply `{ "apply": { "text": "..." } }` or clear `{ "clear": true }`; logged to `governance_events`.
- `scripts/meta-harness/company_os_task_overlay.py` — operator CLI wrapping the same API.
- `src/company_os/agents.rs` — `llm-context` renders **Task-local harness overlay (reversible)** separately from stigmergic handoff notes.


- Keep refinements task-scoped first.
- Allow runtime to load a **task-local policy overlay** between turns, not mid-token.
- Record overlay IDs in task `context_notes` and run metadata.
- Expose active overlay in `llm-context`.

Safety:

- Only reversible overlays.
- No source-file mutation.
- Human approval required for shared/company-wide overlays.

Acceptance:

- A long-running task can adopt a task-local overlay after a weak turn.
- The overlay appears in `llm-context`.
- Removing the overlay restores baseline behavior.

**CLI (shipped):** `scripts/meta-harness/company_os_overlay_from_delta.py` applies a validated `*.filled.json` to `POST .../harness-overlay`. Meta-harness: `--apply-overlay-task-id <uuid>` (or `HSM_COMPANY_OS_OVERLAY_TASK_ID`).

### M5 — PRM / teacher relabel / SFT lane

Goal: separate training lane for model co-learning, not part of live production control.

Implementation:
- `schemas/company_os_rollout_shard.v1.json` — shard contract.
- `scripts/meta-harness/company_os_training_export.py` — JSONL turns → shard files under `runs/company_os_training/`.
- `scripts/meta-harness/company_os_prm_score.py` — heuristic PRM scores per window row.
- `scripts/meta-harness/company_os_training_dry_run.py` — token/label stats without training.


- Define rollout shard schema under `schemas/company_os_rollout_shard.v1.json`.
- Export `company_os_turns.jsonl` windows into teacher-label jobs.
- Add PRM scoring interface:
  - start with heuristic/process scores from `evaluate_turn.py`,
  - later allow frontier teacher labels.
- Store training data under `runs/company_os_training/`.
- Add a dry-run trainer command before real SFT:
  - validates data,
  - computes label distributions,
  - estimates token/compute cost.

Acceptance:

- Can export shards without training.
- Can score/relabel shards without touching production.
- Any SFT adapter requires a held-out Company OS meta-harness eval before use.

## 4. Safety gates

Every stage must preserve these gates:

- **Allowlist gate:** refiner/applier can only target known policy surfaces.
- **Schema gate:** all deltas validate before apply.
- **Review gate:** runtime changes are not automatic.
- **Re-test gate:** candidate policy must be evaluated after apply.
- **Rollback gate:** every promoted policy has a previous version.
- **Ledger gate:** promotions are recorded in artifacts and, when online, Company OS governance events.

## 5. Near-term implementation order

1. Implement `company_os_llm_refiner.py` with schema validation and tests.
2. Add policy file target under `config/company_os_harness/`.
3. Implement `company_os_delta_apply.py` in dry-run/versioned-policy mode.
4. Teach agent-chat to optionally read `HSM_COMPANY_OS_HARNESS_POLICY_PATH`.
5. Add promotion command and smoke workflow.
6. Only then consider task-local online overlays.