# Agent quality cockpit: friendly enterprise observability and evaluation

HSM-II should make agent observability and evaluation feel like a **quality cockpit**, not a specialist monitoring console. Oracle-style enterprise frameworks are right about the core needs: evaluate, trace, report, monitor, compare, and improve. The product opportunity is to make those capabilities understandable to operators, founders, team leads, and enterprise buyers without forcing them to learn evaluation infrastructure first.

The promise:

> Bring your company, choose what good looks like, run the agents, see what happened, understand why, and promote only the changes that measurably improve quality.

## Product stance

Most observability products start with traces, metrics, dashboards, and judges. HSM-II should start with a simpler question:

> Can this agent be trusted with this work?

Everything else supports that answer.

| Enterprise concept | User-facing HSM-II language | What it means |
|--------------------|-----------------------------|---------------|
| Evaluation dataset | Quality checklist | Representative tasks the agent must pass. |
| Eval pack | Company quality pack | Declarative "what good looks like" for one company. |
| Trace | Run story | The step-by-step account of what the agent did. |
| LLM-as-a-judge | Review judge | Optional semantic reviewer, used when deterministic checks are not enough. |
| Metric threshold | Quality gate | The minimum acceptable bar before promotion. |
| Pareto frontier | Best tradeoffs | Candidates that improve quality without unacceptable cost or latency. |
| Observability dashboard | Agent health | Current reliability, cost, latency, and failure patterns. |
| Regression | Got worse | A newer version performs worse than the known-good version. |

The UI and docs can still expose the technical terms for advanced users, but the first path should use plain words.

## What makes it better

HSM-II can be friendlier than a generic enterprise observability suite because it owns the whole loop: company graph, tasks, agents, memory, tools, traces, verifier gates, and promotion records.

| Common enterprise pattern | HSM-II improvement |
|---------------------------|--------------------|
| Start with dashboards full of metrics. | Start with a human-readable verdict: **Ready**, **Needs review**, or **Do not promote**. |
| Users manually map traces to business outcomes. | Eval packs bind every run to company tasks, expected state, tools, memory, and governance. |
| LLM judges become the default quality layer. | Deterministic verifiers run first; judge models are used only where semantic review is needed. |
| Traces are developer-only debugging artifacts. | Run stories explain the agent's plan, tools, evidence, result, and failure reason in operator language. |
| Evaluation is separate from deployment. | Promotion is part of the same workflow: run, compare, shadow, approve, roll back. |
| Enterprise configuration feels like a platform project. | A company starts from templates: "Support agent quality pack," "Sales ops pack," "Engineering DRI pack," then edits plain YAML or UI fields. |
| Reports are static exports. | Reports become operating evidence: which agent improved, what changed, what failed, who approved. |

## User journey

The core product should feel like five steps:

1. **Pick the work:** choose a company, agent, and quality pack.
2. **Run the check:** evaluate draft/current/candidate versions on representative tasks.
3. **Read the verdict:** see score, outcome, cost, latency, and the top failure causes in plain language.
4. **Open the run story:** inspect the steps, tool calls, memory used, handoffs, and final answer.
5. **Improve safely:** accept a suggested fix, rerun the pack, then promote only if gates pass.

This maps to the existing architecture:

```text
Company quality pack
  -> meta_harness.py / evaluate_turn.py
  -> NDJSON run stories + results.json + summary.json
  -> KPI verifiers + optional judge
  -> refiner delta suggestions
  -> promotion record + rollback gate
```

## Product surfaces

### 1. Quality packs

Quality packs are the friendly layer over `schemas/company_eval_pack.v1.json`.

They should be editable in two modes:

- **Simple mode:** task name, persona, prompt, what should happen, pass threshold.
- **Advanced mode:** scorer binding, KPI keys, weights, slice tags, safety gates, promotion rules.

The product should ship starter packs:

| Pack | Use |
|------|-----|
| Support resolution | Did the agent answer accurately, use knowledge, avoid unsafe claims, and route hard cases? |
| Sales operations | Did the agent update CRM-like state, use correct account data, and produce useful follow-up? |
| Engineering DRI | Did the agent inspect the repo, create/update tasks, identify blockers, and cite evidence? |
| Finance operations | Did the agent respect approval gates, avoid unauthorized actions, and produce auditable summaries? |
| HR / policy assistant | Did the agent stay grounded in policy docs and escalate sensitive cases? |

### 2. Agent health

Agent health is the executive and operator view.

Default cards:

- **Trust score:** blended quality, grounding, and completion.
- **Business outcomes:** tasks resolved, routed, created, updated, or blocked correctly.
- **Grounding:** how often answers cite or use real company memory/data.
- **Tool reliability:** correct tool, correct arguments, successful completion.
- **Cost and latency:** median and P95/P99 where available.
- **Safety:** human gates, policy violations, PII/prompt-injection flags when implemented.
- **Trend:** better, worse, or unchanged compared with the last known-good version.

The main rule: every number must have a "show me why" path to a run story.

### 3. Run stories

Run stories translate traces into something a non-observability expert can read.

Each run story should show:

- **Intent:** what the user asked.
- **Plan:** what the agent decided to do.
- **Evidence:** memory, tools, documents, tasks, or APIs used.
- **Actions:** created/updated/routed objects and tool calls.
- **Result:** final answer and post-run state.
- **Quality verdict:** pass/fail/needs review with reasons.
- **Cost and time:** latency, token/cost estimates, tool count.
- **Fix suggestions:** what to change next when it fails.

Advanced users can still open raw NDJSON, spans, token counts, and payloads.

### 4. Failure explanations

The product should avoid "score: 0.62" as the primary explanation. It should say:

- "The agent answered, but did not use company memory."
- "The task was created, but assigned to the wrong owner."
- "The handoff is still pending, so this is not complete."
- "The right tool was used, but required parameters were missing."
- "The answer is plausible, but not grounded in retrieved evidence."
- "Latency exceeded the pack budget."

These map to the failure taxonomy in `eval-pack-and-scoring-contract.md`.

### 5. Promotion and rollback

Promotion is where HSM-II can feel meaningfully safer than normal agent platforms.

Use plain gates:

| Gate | Question |
|------|----------|
| Quality gate | Did the candidate meet the company quality bar? |
| Regression gate | Did anything important get worse? |
| Cost gate | Is the improvement worth the added cost? |
| Latency gate | Is it still fast enough for the workflow? |
| Safety gate | Did it respect human approval and policy boundaries? |
| Evidence gate | Can we explain why it passed? |

Only after those pass should a candidate move to shadow, limited rollout, or full promotion. Every promotion needs a record with before/after metrics, approver, source pack, candidate ID, and rollback rule.

## Enterprise offer

Position this as:

> Enterprise AI agent observability and evaluation for teams that want evidence, not mystery dashboards.

Core offer:

- **Built-in evaluation:** reusable company quality packs, versioned runs, deterministic KPI checks, optional judge review.
- **Built-in tracing:** run stories with raw trace access for developers.
- **Built-in reporting:** agent health, quality trends, cost/latency, failure causes, before/after comparisons.
- **Built-in improvement loop:** refiner suggestions, reruns, promotion gates, rollback records.
- **Built-in governance:** company-scoped data, audit trails, DRI/human approval gates, private deployment path.

Enterprise buyers get:

- Private or self-hosted deployment.
- Tenant isolation and scoped company data.
- Custom quality packs for their workflows.
- Connector review and tool allowlists.
- Security, audit, and retention controls.
- Exportable evidence for model risk, compliance, and operational review.

## Implementation map

### Implemented now

- Company eval packs and schema: `schemas/company_eval_pack.v1.json`.
- Pack loading: `scripts/meta-harness/pack_loader.py`.
- Agent-chat evaluation: `scripts/meta-harness/meta_harness.py` and `evaluate_turn.py`.
- KPI verifiers: `scripts/meta-harness/kpi_verifier.py`.
- Trace/refiner artifacts: `results.json`, per-task NDJSON, `summary.json`, `company_os_turns.jsonl`, and harness deltas.
- Promotion record scaffold: `scripts/meta-harness/company_os_promote_delta.py`.

### Near-term product slice

1. Add weighted aggregation and slice summaries to `meta_harness.py`.
2. Emit failure tags into `summary.json`.
3. Generate a human-readable `run_story.md` per task from NDJSON + KPI verifier output.
4. Add a simple `quality_report.md` per candidate.
5. Surface quality pack runs in Company Console as **Agent Health**.
6. Add before/after comparison view for two candidate IDs.

### Later enterprise slice

- Continuous production monitoring from live traces.
- Shadow-run routing and promotion service.
- Human annotation queue for uncertain cases.
- Judge-model calibration and audit export.
- Retention policy for eval traces and customer data.
- Portfolio-level views across companies, products, and agent families.

## Copy blocks

Short:

> HSM-II gives enterprises a friendly quality cockpit for AI agents: define what good looks like, run agents against real business tasks, inspect the run story, and promote only the versions that improve quality, cost, latency, and safety.

Medium:

> Most agent observability tools show traces and metrics. HSM-II turns them into operating decisions. Company quality packs define business outcomes, grounding, tool behavior, and safety gates. The meta-harness runs agents, scores every trace, explains failures in plain language, suggests improvements, and records promotions with rollback evidence.

Enterprise:

> HSM-II provides built-in AI agent observability and evaluation for enterprise workflows: design-time evaluation packs, deterministic KPI verifiers, optional semantic judges, step-by-step run stories, quality and cost reporting, trace-backed failure taxonomy, and governed promotion from offline validation to shadow rollout. It is built for teams that need agent performance they can explain, improve, and audit.