How HSM-II implements field coordination (stigmergy, coherence, DRIs, ratchets)
This note maps the biology-inspired framing (shared field, traces, morphogenesis, collective competence, failure of coordination, niche as memory) onto concrete mechanisms in this repository: simulation stack, Company OS ledger, and eval harness tooling.
1. The shared field (stigmergy, “engrams,” extended mind)
Hypergraph / morphogenesis stack
The medium is explicit: HyperStigmergicMorphogenesis (src/hyper_stigmergy.rs) holds agents, vertices, and hyperedges (weighted, n-ary links). Agents coordinate through the graph—not through a single process that owns all state in RAM.
Traces and policy memory
stigmergic_policy (src/stigmergic_policy.rs) defines structured StigmergicTrace, RoutingDirective, PolicyShift, and StigmergicMemory. These are durable signals in the field that later turns can read—software analogues of “stress written into tissue,” as artifacts with explicit types.
Company OS as canonical operational field
For product / org use, the canonical shared medium is Postgres Company OS: company_memory_entries, task context_notes, governance_events, capability_refs, and the composed read model GET …/tasks/:id/llm-context. Agents are steered by what landed in the ledger, not by chat as system of record.
Contracts:
One-line product rule (from alignment doc): The intelligence layer is the composer; Company OS is the constitution and ledger; DRIs are the circuit breakers and outcome owners at the edge.
Coordination contract for models, harnesses, and systems
The repo now has enough moving parts that every "smart" component needs to know which surface is authoritative. Use this as the shared contract:
| Component | Reads from | Writes to | Must not do |
|---|---|---|---|
| Live models / agents | GET /api/company/tasks/:id/llm-context, company_memory_search, task attachments, linked capability_refs, workforce profile. | Task context_notes, company_memory_entries, task state, run feedback, requires_human / DRI escalation routes. | Treat chat history, Paperclip demo memory, or local scratch files as the company ledger. |
| Company OS runtime | Postgres rows keyed by company_id: companies, tasks, goals, DRIs, memory, spend, governance, runs. | The same Postgres graph plus telemetry events and composed read models. | Fork operational truth into a second task/goal registry without sync. |
| External intelligence / Paperclip-style services | Signals, external tools, optional Company OS APIs. | Upsert/sync into Company OS: goals, tasks, DRIs, governance events, memory rows. | Present a parallel dashboard as authoritative if it cannot point at company_id + API path. |
| Eval harnesses | Benchmark tasks, HsmRunnerConfig, artifact trees, turns_hsm.jsonl, turns_baseline.jsonl, optional traces. | runs/, runs_index.jsonl, candidate result JSON, best_config.json, harness_delta_<n>.json. | Assume a promoted eval config changes hsm_console, personal_agent, or the live bot until an explicit integration maps the fields. |
| Company OS agent-chat harness | Live Next + hsm_console stream/JSON turns, results.json, results/*.ndjson, task state, worker telemetry. | candidate/refiner/company_os_turns.jsonl and company_os_harness_delta_<n>.json review artifacts. | Auto-edit live prompt policy, skill dispatch, or worker guards without an explicit review/re-test promotion. |
| Refiner / outer loop | Recent eval traces and indexed run history. | Harness deltas, proposer context, candidate configs. | Auto-promote deltas into production behavior without a re-run gate. |
| DRIs / humans | Intelligence summary, task graph, governance events, context notes, evidence from runs. | Decisions, overrides, requires_human resolution, DRI assignments, policy/governance rows. | Let high-stakes model decisions finalize only in chat or transient worker output. |
The important invariant is simple: models get their situational awareness from a composed read model; harnesses improve eval behavior through artifacts; runtime state changes land in Company OS; humans/DRIs own irreversible or ambiguous outcomes.
2. Morphogenesis and coherence (shape without a puppet master)
The hyper_stigmergy layer treats structure as something that evolves under local actions (e.g. linking, decay, embeddings, clusters).
Coherence is exposed as a global scalar on the hypergraph (total hyperedge weight normalized by agent count) and is used in collective scoring (e.g. JW in compute_jw: mixes individual action, collective reuse of work, coherence improvement, and stability). That is an engineering analogue of “local sculpting + field-level pressure,” not a literal bioelectric model.
3. Collective intelligence and federation (larger “light cone”)
Multi-system topology appears in meta_graph (src/meta_graph.rs): shared meta-hypergraph (federation integration point, “H-star” in code comments), promoted edges, conflict mediation.
Councils consume graph-backed evidence (including traces and directives) so high-stakes decisions are field-grounded where wired (src/council/, evidence from property graph / traces in hyper_stigmergy).
4. When coordination breaks (“cancer-like” defection)
Operationally, failure looks like automation or agents ignoring the ledger: parallel truths, skipping requires_human, side-channel plans that never become rows.
Counterweights in HSM-II:
requires_human,dri_assignments,governance_events- Rule: intelligence proposes; Postgres + humans dispose (see alignment doc)
Provenance for governance (runs, traces, events) coexists with shared memory for routine work: you keep attribution where accountability needs it, even when day-to-day agent context is summarized.
5. Niche construction and memory (environment as memory)
Org-level “niche” is shaped by:
company_memory,companies.context_markdown- Visions / playbooks (see playbooks, projects, and visions)
skills/and taskcapability_refsso procedures become repeatable, queryable structure in the field
6. Competency ratchet (eval, meta-harness, outer loop, refiner stub)
| Piece | Role |
|---|---|
hsm-eval | Inner eval run; writes turns_hsm.jsonl and related artifacts under runs/ or --artifacts |
hsm_meta_harness | Outer search over harness configs; leaderboard / promote path |
hsm_outer_loop | Index / query runs; proposer context for agents |
scripts/refiner-jsonl-pass.py | Deterministic F-step pass on turns_hsm.jsonl → harness_delta_<n>.json (no LLM); spec in EVAL_AND_META_HARNESS.md |
scripts/meta-harness/company_os_refiner_pass.py | Deterministic Company OS agent-chat pass: results.json + results/*.ndjson → company_os_turns.jsonl + company_os_harness_delta_<n>.json |
The refiner scripts are small offline slices of “periodic harness review on traces,” not the live product brain and not a full implementation of Continual Harness. They follow the paper’s W/F trace gate shape, but today they write placeholder delta JSON for review instead of applying online edits to a running harness. Promoted harness JSON remains eval-side until explicitly wired into a runtime (see EVAL doc § promoted config vs production).
Ratchet gates
For a harness change to affect live behavior, require all three gates:
- Trace gate: the change is grounded in an artifact (
turns_hsm.jsonl,hsm_trace.jsonl, run feedback, or a task-linked failure note), not an unobserved preference. - Re-test gate: the candidate config or delta is re-run with
hsm-eval/ the Company OS agent-chat harness and produces comparable evidence. - Runtime mapping gate: a human or integration explicitly maps the winning eval field into the target runtime (
hsm_console,personal_agent, worker prompt policy, model routing, memory policy, tool allowlist, or Company OS API behavior).
Until gate 3 happens, eval wins are knowledge and candidate policy, not production state.
7. Bottom line
HSM-II implements field-mediated coordination by making Postgres + hypergraph + traces + composed llm-context the shared interpretable medium; coherence / JW / councils supply collective pressure terms; DRIs + governance flags restore binding constraints when local optimization diverges; and eval + meta-harness + refiner support a competency ratchet on harness behavior—without collapsing “intelligence,” “ledger,” and “eval” into one undocumented system.
Related references
- Eval, meta-harness, and outer loop
- Stigmergic DKS entities (second-order ecology on hyperedges)
- Continual Harness (paper PDF) — online harness refinement; JSONL F-step loop documented in EVAL doc
- Ax + harness evolution stack — @ax-llm/ax, claude-harness, both papers mapped to code
- FlashEvolve (paper PDF) — async stage orchestration for GEPA/meta-harness throughput (roadmap)