# How HSM-II implements field coordination (stigmergy, coherence, DRIs, ratchets)

This note maps the **biology-inspired framing** (shared field, traces, morphogenesis, collective competence, failure of coordination, niche as memory) onto **concrete mechanisms in this repository**: simulation stack, Company OS ledger, and eval harness tooling.

---

## 1. The shared field (stigmergy, “engrams,” extended mind)

### Hypergraph / morphogenesis stack

The medium is explicit: **`HyperStigmergicMorphogenesis`** (`src/hyper_stigmergy.rs`) holds agents, vertices, and **hyperedges** (weighted, n-ary links). Agents coordinate **through the graph**—not through a single process that owns all state in RAM.

### Traces and policy memory

**`stigmergic_policy`** (`src/stigmergic_policy.rs`) defines structured **`StigmergicTrace`**, **`RoutingDirective`**, **`PolicyShift`**, and **`StigmergicMemory`**. These are **durable signals in the field** that later turns can read—software analogues of “stress written into tissue,” as **artifacts** with explicit types.

### Company OS as canonical operational field

For product / org use, the canonical shared medium is **Postgres Company OS**: **`company_memory_entries`**, task **`context_notes`**, **`governance_events`**, **`capability_refs`**, and the composed read model **`GET …/tasks/:id/llm-context`**. Agents are steered by **what landed in the ledger**, not by chat as system of record.

Contracts:

- [Company world model and intelligence](./company-os/world-model-and-intelligence.md)
- [Intelligence layer, DRIs, and alignment](./company-os/intelligence-layer-dri-alignment.md)

**One-line product rule (from alignment doc):** *The intelligence layer is the composer; Company OS is the constitution and ledger; DRIs are the circuit breakers and outcome owners at the edge.*

### Coordination contract for models, harnesses, and systems

The repo now has enough moving parts that every "smart" component needs to know which surface is authoritative. Use this as the shared contract:

| Component | Reads from | Writes to | Must not do |
|-----------|------------|-----------|-------------|
| **Live models / agents** | `GET /api/company/tasks/:id/llm-context`, `company_memory_search`, task attachments, linked `capability_refs`, workforce profile. | Task `context_notes`, `company_memory_entries`, task state, run feedback, `requires_human` / DRI escalation routes. | Treat chat history, Paperclip demo memory, or local scratch files as the company ledger. |
| **Company OS runtime** | Postgres rows keyed by `company_id`: companies, tasks, goals, DRIs, memory, spend, governance, runs. | The same Postgres graph plus telemetry events and composed read models. | Fork operational truth into a second task/goal registry without sync. |
| **External intelligence / Paperclip-style services** | Signals, external tools, optional Company OS APIs. | Upsert/sync into Company OS: goals, tasks, DRIs, governance events, memory rows. | Present a parallel dashboard as authoritative if it cannot point at `company_id` + API path. |
| **Eval harnesses** | Benchmark tasks, `HsmRunnerConfig`, artifact trees, `turns_hsm.jsonl`, `turns_baseline.jsonl`, optional traces. | `runs/`, `runs_index.jsonl`, candidate result JSON, `best_config.json`, `harness_delta_<n>.json`. | Assume a promoted eval config changes `hsm_console`, `personal_agent`, or the live bot until an explicit integration maps the fields. |
| **Company OS agent-chat harness** | Live Next + `hsm_console` stream/JSON turns, `results.json`, `results/*.ndjson`, task state, worker telemetry. | `candidate/refiner/company_os_turns.jsonl` and `company_os_harness_delta_<n>.json` review artifacts. | Auto-edit live prompt policy, skill dispatch, or worker guards without an explicit review/re-test promotion. |
| **Refiner / outer loop** | Recent eval traces and indexed run history. | Harness deltas, proposer context, candidate configs. | Auto-promote deltas into production behavior without a re-run gate. |
| **DRIs / humans** | Intelligence summary, task graph, governance events, context notes, evidence from runs. | Decisions, overrides, `requires_human` resolution, DRI assignments, policy/governance rows. | Let high-stakes model decisions finalize only in chat or transient worker output. |

The important invariant is simple: **models get their situational awareness from a composed read model; harnesses improve eval behavior through artifacts; runtime state changes land in Company OS; humans/DRIs own irreversible or ambiguous outcomes.**

---

## 2. Morphogenesis and coherence (shape without a puppet master)

The **`hyper_stigmergy`** layer treats structure as something that **evolves under local actions** (e.g. linking, decay, embeddings, clusters).

**Coherence** is exposed as a **global scalar** on the hypergraph (total hyperedge weight normalized by agent count) and is used in **collective scoring** (e.g. **JW** in `compute_jw`: mixes individual action, collective reuse of work, coherence improvement, and stability). That is an engineering analogue of “local sculpting + field-level pressure,” not a literal bioelectric model.

---

## 3. Collective intelligence and federation (larger “light cone”)

**Multi-system topology** appears in **`meta_graph`** (`src/meta_graph.rs`): shared **meta-hypergraph** (federation integration point, “H-star” in code comments), promoted edges, conflict mediation.

**Councils** consume **graph-backed evidence** (including traces and directives) so high-stakes decisions are **field-grounded** where wired (`src/council/`, evidence from property graph / traces in `hyper_stigmergy`).

---

## 4. When coordination breaks (“cancer-like” defection)

Operationally, failure looks like **automation or agents ignoring the ledger**: parallel truths, skipping **`requires_human`**, side-channel plans that never become rows.

Counterweights in HSM-II:

- **`requires_human`**, **`dri_assignments`**, **`governance_events`**
- Rule: **intelligence proposes; Postgres + humans dispose** (see alignment doc)

**Provenance** for governance (runs, traces, events) coexists with **shared memory** for routine work: you keep attribution where **accountability** needs it, even when day-to-day agent context is **summarized**.

---

## 5. Niche construction and memory (environment as memory)

Org-level “niche” is shaped by:

- **`company_memory`**, **`companies.context_markdown`**
- **Visions / playbooks** (see [playbooks, projects, and visions](./company-os/playbooks-projects-and-visions.md))
- **`skills/`** and task **`capability_refs`** so procedures become **repeatable, queryable structure** in the field

---

## 6. Competency ratchet (eval, meta-harness, outer loop, refiner stub)

| Piece | Role |
|-------|------|
| **`hsm-eval`** | Inner eval run; writes **`turns_hsm.jsonl`** and related artifacts under `runs/` or `--artifacts` |
| **`hsm_meta_harness`** | Outer search over harness configs; leaderboard / promote path |
| **`hsm_outer_loop`** | Index / query runs; proposer context for agents |
| **`scripts/refiner-jsonl-pass.py`** | Deterministic **F-step** pass on **`turns_hsm.jsonl`** → **`harness_delta_<n>.json`** (no LLM); spec in [EVAL_AND_META_HARNESS.md](./EVAL_AND_META_HARNESS.md) |
| **`scripts/meta-harness/company_os_refiner_pass.py`** | Deterministic Company OS agent-chat pass: **`results.json` + `results/*.ndjson`** → **`company_os_turns.jsonl`** + **`company_os_harness_delta_<n>.json`** |

The refiner scripts are **small offline slices** of “periodic harness review on traces,” not the live product brain and not a full implementation of Continual Harness. They follow the paper’s **W/F trace gate shape**, but today they write placeholder delta JSON for review instead of applying online edits to a running harness. Promoted harness JSON remains **eval-side** until explicitly wired into a runtime (see EVAL doc § promoted config vs production).

### Ratchet gates

For a harness change to affect live behavior, require all three gates:

1. **Trace gate:** the change is grounded in an artifact (`turns_hsm.jsonl`, `hsm_trace.jsonl`, run feedback, or a task-linked failure note), not an unobserved preference.
2. **Re-test gate:** the candidate config or delta is re-run with `hsm-eval` / the Company OS agent-chat harness and produces comparable evidence.
3. **Runtime mapping gate:** a human or integration explicitly maps the winning eval field into the target runtime (`hsm_console`, `personal_agent`, worker prompt policy, model routing, memory policy, tool allowlist, or Company OS API behavior).

Until gate 3 happens, eval wins are **knowledge and candidate policy**, not production state.

---

## 7. Bottom line

HSM-II implements **field-mediated coordination** by making **Postgres + hypergraph + traces + composed `llm-context`** the **shared interpretable medium**; **coherence / JW / councils** supply **collective pressure terms**; **DRIs + governance flags** restore **binding constraints** when local optimization diverges; and **eval + meta-harness + refiner** support a **competency ratchet** on harness behavior—without collapsing “intelligence,” “ledger,” and “eval” into one undocumented system.

---

## Related references

- [Eval, meta-harness, and outer loop](./EVAL_AND_META_HARNESS.md)
- [Stigmergic DKS entities](../src/dks/stigmergic_entity.rs) (second-order ecology on hyperedges)
- [Continual Harness (paper PDF)](https://arxiv.org/pdf/2605.09998) — online harness refinement; JSONL F-step loop documented in EVAL doc
- [Ax + harness evolution stack](./company-os/ax-harness-evolution-stack.md) — [@ax-llm/ax](https://github.com/ax-llm/ax), claude-harness, both papers mapped to code
- [FlashEvolve (paper PDF)](https://arxiv.org/pdf/2605.08520) — async stage orchestration for GEPA/meta-harness throughput (roadmap)