5 · Assure

Observation tells you what the system is doing. Assurance answers the harder questions — the ones a risk owner, an incident review, or an auditor asks:

Question	Answered by
What did it actually do, task by task, and what did each task cost?	The run ledger
Has it stopped working?	Liveness
Is it behaving differently than it used to — and why?	Behavioural drift
What exactly was it allowed to do on the day that run executed?	Contracts
Is it being used by whom, and for what, it was declared for?	Conformance
Is it healthy right now?	Worst-of health
Can you hand someone the evidence?	The Assurance Report

Everything here attaches to the AI System — the same entity you created in Deploy — and most of it feeds itself from traffic you’re already sending. This page is the operational walkthrough; the AI System Assurance section explains the concepts from first principles and holds the full mechanics reference, one page per signal.

What assurance honestly claims

Brutor detects behavioural change against the system’s own history — it does not grade answer quality; there is no ground truth at the proxy. Liveness means “it stopped calling through the gateway”, not process uptime. When no cause lines up, findings say unexplained rather than inventing one. The honesty is the point: evidence you can’t trust the edges of isn’t evidence.

Day 0 — three switches

Assurance mostly runs itself, but three things want a decision the day the system goes live:

1. Set the liveness expectation — continuous, scheduled, sporadic or on_demand (right for support-copilot; a nightly pipeline wants scheduled). This decision was already required by the gate; the detectors — silence, missed schedule, volume collapse, stall, hard-fail cliff — run against it, and Brutor can learn the cadence from history (sized on the p95 inter-run gap) instead of you guessing.

2. Check the run ledger is filling — AI Estate → AI Systems → your system → Runs. If runs look like single requests, your run header isn’t reaching the gateway. Fix it now: every assurance signal downstream is built on runs.

3. Decide on automatic snapshots — assurance_capture_interval_hours on the tenant captures immutable assurance reports on a schedule (off by default). If this system will ever face an auditor, turn it on: continuous post-market evidence produced by a machine beats someone remembering to click in December.

The first weeks — learning, then drift

For its first ~50 baseline-eligible runs the system is learning: no drift alerts, and the UI says “still learning this system’s normal” instead of showing an empty list that reads as a clean bill of health. Once baselines exist, drift detection compares current behaviour to them — robust statistics, sustained-change requirements (one bad afternoon is not drift) — and, crucially, ships the cause, not a p-value:

action_count up 180% since 14:00 on the 14th; the provider shipped a new point release of the underlying model at 13:52.

The gateway is the only component that sees both your traffic and the model registry, so it’s the only thing that can make that correlation. Candidate causes are checked in order — model change, tool change, composition change, input drift — and terminate at unexplained when nothing lines up. Severity ranks by consequence (effect size × volume × business impact), not statistical confidence.

Triage each finding: acknowledge, resolve, or not drift — the last one deliberately separate, because it’s evidence about the detector, and a detector that can’t be told it was wrong gets muted within a month.

Close the loop — response policies

Detection can act. A response policy arms an automatic reaction on the system:

on:
  drift_class: behavioural
  severity_at_least: high
then:
  - set_autonomy: approval_required
require_human_to_restore: true

The runaway-refund scenario ends differently with this armed: cost-per-run drifts critical at 02:00, autonomy tightens to approval_required, and the 03:00 refund parks in an inbox instead of executing. The rails are deliberate — a policy never loosens, suspension is opt-in with a second acknowledgement, one action per cooldown, and restore is always human (automatic restore flaps). Every action and every considered-but-not-fired decision is recorded, as actor system:drift-response, with the blast radius previewed before you arm.

Contracts and conformance — prove the permissions

The contract you minted at deploy keeps working after launch:

• Config changes → mint a new version; the stored diff separates widened from narrowed, and the approver sees exactly the diff they approved, forever.
• Conformance → because contracts are generated from live config, “is this system running configuration nobody approved?” is computable — and stated plainly on the Contracts panel.
• Declared vs observed → an undeclared client calling your system, a declared client that never calls, a granted resource never used (least-privilege finding): all reported once there’s enough traffic to mean something.

Before shipping a config change, replay the candidate contract against a stratified sample of real recorded runs — rarest-first sampling, so the suite includes the exhausted runs and rare tools a happy-path sample misses. outcome_changed (an action that used to succeed would now be denied) fails the gate; a new approval requirement is reported as policy_changed, stricter-not-broken.

Health — one number you can decompose

health = min(liveness, behaviour, reliability, cost, conformance)

Worst-of, never an average — a system that is cheap, fast and dead is not 80% healthy. Only healthy renders green; learning and unknown are amber because they mean unmeasured, and unmeasured is exactly what an optimistic dashboard rounds up. The chain-integrity percentage sits beside the score so you know what it’s worth.

The Assurance Report

One artifact per system, answering every question in the table at the top with evidence: identity and declared intent, contract history with approvers and hashes, drift events with causes and responses taken, run outcomes and cost per completed run, liveness record, replay results, conformance. The engineer opens it at 09:00; the auditor exports it in December — same artifact.

Snapshots are immutable (evidence that changes when the system does isn’t evidence), and every report ends with what it does not cover — chain integrity below 100%, guardrails not re-evaluated in replay, clients that never traversed the gateway. Captured reports feed ISO 42001 control coverage (A.6.2.4 performance evaluation, A.6.2.7 post-market monitoring) automatically.

curl http://localhost:5050/v1/admin/ai-systems/$G/assurance \
  -H "Authorization: Bearer $ADMIN_TOKEN"          # the live report
curl -X POST http://localhost:5050/v1/admin/ai-systems/$G/assurance \
  -H "Authorization: Bearer $ADMIN_TOKEN"          # capture an immutable snapshot

Tamper evidence — proving the records themselves

Everything above reasons over proxy_logs. That raises an obvious question an auditor will ask: how do you know those rows say what they said when they were written?

Every audit row carries a SHA-256 covering its own contents and its predecessor’s hash, so altering any row breaks every hash after it. Chains are per proxy process — a restart begins a new one, which is expected. Batches of rows are signed with Ed25519 into audit_chain_checkpoints, because hashes alone only detect edits: anyone able to rewrite the rows could recompute the chain. A signature is what makes that require a key.

Verify from System → Audit Chains in the Admin Console, or:

curl -X POST "http://localhost:5050/v1/admin/audit/chains/$CHAIN/verify" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Six verdicts, because “not verified” covers situations you would act on differently:

Verdict	What it claims
verified	Hashes link and a checkpoint signed by a trusted key covers them. The only verdict asserting integrity.
unattested	Hashes link; nothing signed them. No signing key configured — see BRUTOR_AUDIT_SIGNING_KEY.
gap	Sequence numbers missing: rows were deleted.
broken	A row no longer hashes to its stored value.
forged	A signature verifies against a key this deployment never trusted. Ranked worst — a hash break can be corruption, an unauthorised attestation is deliberate.
unchained	No chain columns. Rows written before tamper-evidence was enabled — not a pass and not a failure.

Consistent with the rest of assurance: absence of evidence is never reported as evidence of integrity, and a check that could not run returns an error rather than a clean result.

Scheduled audit exports to S3 carry the same property outward. Each export writes a <object>.manifest.json sidecar with a SHA-256 over the delivered bytes — so a recipient verifies with sha256sum alone — signed by the same key. The manifest also records chain_columns_included, because a field allowlist that strips row_hash silently makes an export unverifiable and the recipient cannot tell from the file.

End of the journey — and the loop back

Retirement is a gate too: active → deprecated requires naming a successor (or stating there is none), and deprecated → retired requires 30 quiet days. Nothing simply disappears with its history.

More often, assurance sends you backwards around the loop: a drift finding becomes a governance change (Govern), tested by replay, minted as a contract version, observed (Observe) — one AI System, one thread of evidence, the whole way around.

How assurance worksThe concepts from first principles — the four guarantees and the full cycle, with a detail page per signal.

Assurance APIEvery endpoint referenced on this page.

Fleet viewZoom out: every AI System, asset and identity in the tenant, with governance gaps.