Agent quality cockpit: friendly enterprise observability and evaluation

HSM-II should make agent observability and evaluation feel like a quality cockpit, not a specialist monitoring console. Oracle-style enterprise frameworks are right about the core needs: evaluate, trace, report, monitor, compare, and improve. The product opportunity is to make those capabilities understandable to operators, founders, team leads, and enterprise buyers without forcing them to learn evaluation infrastructure first.

The promise:

Bring your company, choose what good looks like, run the agents, see what happened, understand why, and promote only the changes that measurably improve quality.

Product stance

Most observability products start with traces, metrics, dashboards, and judges. HSM-II should start with a simpler question:

Can this agent be trusted with this work?

Everything else supports that answer.

Enterprise concept	User-facing HSM-II language	What it means
Evaluation dataset	Quality checklist	Representative tasks the agent must pass.
Eval pack	Company quality pack	Declarative "what good looks like" for one company.
Trace	Run story	The step-by-step account of what the agent did.
LLM-as-a-judge	Review judge	Optional semantic reviewer, used when deterministic checks are not enough.
Metric threshold	Quality gate	The minimum acceptable bar before promotion.
Pareto frontier	Best tradeoffs	Candidates that improve quality without unacceptable cost or latency.
Observability dashboard	Agent health	Current reliability, cost, latency, and failure patterns.
Regression	Got worse	A newer version performs worse than the known-good version.

The UI and docs can still expose the technical terms for advanced users, but the first path should use plain words.

What makes it better

HSM-II can be friendlier than a generic enterprise observability suite because it owns the whole loop: company graph, tasks, agents, memory, tools, traces, verifier gates, and promotion records.

Common enterprise pattern	HSM-II improvement
Start with dashboards full of metrics.	Start with a human-readable verdict: Ready, Needs review, or Do not promote.
Users manually map traces to business outcomes.	Eval packs bind every run to company tasks, expected state, tools, memory, and governance.
LLM judges become the default quality layer.	Deterministic verifiers run first; judge models are used only where semantic review is needed.
Traces are developer-only debugging artifacts.	Run stories explain the agent's plan, tools, evidence, result, and failure reason in operator language.
Evaluation is separate from deployment.	Promotion is part of the same workflow: run, compare, shadow, approve, roll back.
Enterprise configuration feels like a platform project.	A company starts from templates: "Support agent quality pack," "Sales ops pack," "Engineering DRI pack," then edits plain YAML or UI fields.
Reports are static exports.	Reports become operating evidence: which agent improved, what changed, what failed, who approved.

User journey

The core product should feel like five steps:

1. Pick the work: choose a company, agent, and quality pack.
2. Run the check: evaluate draft/current/candidate versions on representative tasks.
3. Read the verdict: see score, outcome, cost, latency, and the top failure causes in plain language.
4. Open the run story: inspect the steps, tool calls, memory used, handoffs, and final answer.
5. Improve safely: accept a suggested fix, rerun the pack, then promote only if gates pass.

This maps to the existing architecture:

Company quality pack
  -> meta_harness.py / evaluate_turn.py
  -> NDJSON run stories + results.json + summary.json
  -> KPI verifiers + optional judge
  -> refiner delta suggestions
  -> promotion record + rollback gate

Product surfaces

1. Quality packs

Quality packs are the friendly layer over schemas/company_eval_pack.v1.json.

They should be editable in two modes:

• Simple mode: task name, persona, prompt, what should happen, pass threshold.
• Advanced mode: scorer binding, KPI keys, weights, slice tags, safety gates, promotion rules.

The product should ship starter packs:

Pack	Use
Support resolution	Did the agent answer accurately, use knowledge, avoid unsafe claims, and route hard cases?
Sales operations	Did the agent update CRM-like state, use correct account data, and produce useful follow-up?
Engineering DRI	Did the agent inspect the repo, create/update tasks, identify blockers, and cite evidence?
Finance operations	Did the agent respect approval gates, avoid unauthorized actions, and produce auditable summaries?
HR / policy assistant	Did the agent stay grounded in policy docs and escalate sensitive cases?

2. Agent health

Agent health is the executive and operator view.

Default cards:

• Trust score: blended quality, grounding, and completion.
• Business outcomes: tasks resolved, routed, created, updated, or blocked correctly.
• Grounding: how often answers cite or use real company memory/data.
• Tool reliability: correct tool, correct arguments, successful completion.
• Cost and latency: median and P95/P99 where available.
• Safety: human gates, policy violations, PII/prompt-injection flags when implemented.
• Trend: better, worse, or unchanged compared with the last known-good version.

The main rule: every number must have a "show me why" path to a run story.

3. Run stories

Run stories translate traces into something a non-observability expert can read.

Each run story should show:

• Intent: what the user asked.
• Plan: what the agent decided to do.
• Evidence: memory, tools, documents, tasks, or APIs used.
• Actions: created/updated/routed objects and tool calls.
• Result: final answer and post-run state.
• Quality verdict: pass/fail/needs review with reasons.
• Cost and time: latency, token/cost estimates, tool count.
• Fix suggestions: what to change next when it fails.

Advanced users can still open raw NDJSON, spans, token counts, and payloads.

4. Failure explanations

The product should avoid "score: 0.62" as the primary explanation. It should say:

• "The agent answered, but did not use company memory."
• "The task was created, but assigned to the wrong owner."
• "The handoff is still pending, so this is not complete."
• "The right tool was used, but required parameters were missing."
• "The answer is plausible, but not grounded in retrieved evidence."
• "Latency exceeded the pack budget."

These map to the failure taxonomy in eval-pack-and-scoring-contract.md.

5. Promotion and rollback

Promotion is where HSM-II can feel meaningfully safer than normal agent platforms.

Use plain gates:

Gate	Question
Quality gate	Did the candidate meet the company quality bar?
Regression gate	Did anything important get worse?
Cost gate	Is the improvement worth the added cost?
Latency gate	Is it still fast enough for the workflow?
Safety gate	Did it respect human approval and policy boundaries?
Evidence gate	Can we explain why it passed?

Only after those pass should a candidate move to shadow, limited rollout, or full promotion. Every promotion needs a record with before/after metrics, approver, source pack, candidate ID, and rollback rule.

Enterprise offer

Position this as:

Enterprise AI agent observability and evaluation for teams that want evidence, not mystery dashboards.

Core offer:

• Built-in evaluation: reusable company quality packs, versioned runs, deterministic KPI checks, optional judge review.
• Built-in tracing: run stories with raw trace access for developers.
• Built-in reporting: agent health, quality trends, cost/latency, failure causes, before/after comparisons.
• Built-in improvement loop: refiner suggestions, reruns, promotion gates, rollback records.
• Built-in governance: company-scoped data, audit trails, DRI/human approval gates, private deployment path.

Enterprise buyers get:

• Private or self-hosted deployment.
• Tenant isolation and scoped company data.
• Custom quality packs for their workflows.
• Connector review and tool allowlists.
• Security, audit, and retention controls.
• Exportable evidence for model risk, compliance, and operational review.

Implementation map

Implemented now

• Company eval packs and schema: schemas/company_eval_pack.v1.json.
• Pack loading: scripts/meta-harness/pack_loader.py.
• Agent-chat evaluation: scripts/meta-harness/meta_harness.py and evaluate_turn.py.
• KPI verifiers: scripts/meta-harness/kpi_verifier.py.
• Trace/refiner artifacts: results.json, per-task NDJSON, summary.json, company_os_turns.jsonl, and harness deltas.
• Promotion record scaffold: scripts/meta-harness/company_os_promote_delta.py.

Near-term product slice

1. Add weighted aggregation and slice summaries to meta_harness.py.
2. Emit failure tags into summary.json.
3. Generate a human-readable run_story.md per task from NDJSON + KPI verifier output.
4. Add a simple quality_report.md per candidate.
5. Surface quality pack runs in Company Console as Agent Health.
6. Add before/after comparison view for two candidate IDs.

Later enterprise slice

• Continuous production monitoring from live traces.
• Shadow-run routing and promotion service.
• Human annotation queue for uncertain cases.
• Judge-model calibration and audit export.
• Retention policy for eval traces and customer data.
• Portfolio-level views across companies, products, and agent families.

Copy blocks

Short:

HSM-II gives enterprises a friendly quality cockpit for AI agents: define what good looks like, run agents against real business tasks, inspect the run story, and promote only the versions that improve quality, cost, latency, and safety.

Medium:

Most agent observability tools show traces and metrics. HSM-II turns them into operating decisions. Company quality packs define business outcomes, grounding, tool behavior, and safety gates. The meta-harness runs agents, scores every trace, explains failures in plain language, suggests improvements, and records promotions with rollback evidence.

Enterprise:

HSM-II provides built-in AI agent observability and evaluation for enterprise workflows: design-time evaluation packs, deterministic KPI verifiers, optional semantic judges, step-by-step run stories, quality and cost reporting, trace-backed failure taxonomy, and governed promotion from offline validation to shadow rollout. It is built for teams that need agent performance they can explain, improve, and audit.