5 · Assure

Observation tells you what the system is doing. Assurance answers the harder questions — the ones a risk owner, an incident review, or an auditor asks:

Question	Answered by
What did it actually do, task by task, and what did each task cost?	The run ledger
Has it stopped working?	Liveness
Is it behaving differently than it used to — and why?	Behavioural drift
What exactly was it allowed to do on the day that run executed?	Contracts
Is it being used by whom, and for what, it was declared for?	Conformance
Is it healthy right now?	Worst-of health
Can you hand someone the evidence?	The Assurance Report

Everything here attaches to the AI System — the same entity you created in Deploy — and most of it feeds itself from traffic you’re already sending. This page is the operational walkthrough; the AI System Assurance section explains the concepts from first principles and holds the full mechanics reference, one page per signal.

Day 0 — three switches

Assurance mostly runs itself, but three things want a decision the day the system goes live:

Set the liveness expectation — continuous, scheduled, sporadic or on_demand (right for support-copilot; a nightly pipeline wants scheduled), on the system’s Signals → Liveness card. This decision was already required by the gate; the detectors — silence, missed schedule, volume collapse, stall, hard-fail cliff — run against it, and Suggest a cadence from observed runs sizes it from history (the p95 inter-run gap) instead of you guessing.
Check the run ledger is filling — AI Estate → AI Systems → your system → Runs. If runs look like single requests, your run header isn’t reaching the gateway. Fix it now: every assurance signal downstream is built on runs. If part of the system genuinely cannot route through the gateway, the OpenTelemetry GenAI inlet can report those spans into the ledger — they arrive marked reported, never witnessed, and the report says what share of the evidence that is.
Decide on automatic snapshots — assurance_capture_interval_hours on the tenant captures immutable assurance reports on a schedule (off by default); captured ones are listed under the system’s Assurance → Snapshots. If this system will ever face an auditor, turn it on: continuous post-market evidence produced by a machine beats someone remembering to click in December.

The first weeks — learning, then drift

For its first ~50 baseline-eligible runs the system is learning: no drift alerts, and the UI says “still learning this system’s normal” instead of showing an empty list that reads as a clean bill of health. Once baselines exist, drift detection compares current behaviour to them — robust statistics, sustained-change requirements (one bad afternoon is not drift) — and, crucially, ships the cause, not a p-value:

action_count up 180% since 14:00 on the 14th; the provider shipped a new point release of the underlying model at 13:52.

The gateway is the only component that sees both your traffic and the model registry, so it’s the only thing that can make that correlation. Candidate causes are checked in order — model change, tool change, composition change, input drift — and terminate at unexplained when nothing lines up. Severity ranks by consequence (effect size × volume × business impact), not statistical confidence.

Triage each finding on the Drift tab: acknowledge, resolve, or not drift — the last one deliberately separate, because it’s evidence about the detector, and a detector that can’t be told it was wrong gets muted within a month. The same finding is also raised in Mission Control → Inbox, where resolving it records a disposition (true_positive, false_positive, expected, fixed) and a note in an append-only trail — the evidence that someone accountable looked. Leave a critical item open past the review SLA (7 days by default) and the system’s own health degrades: review debt is a health fact, not a backlog metric.

Close the loop — response policies

Detection can act. A response policy (Drift → Add a response policy) arms an automatic reaction on the system:

on:
  drift_class: behavioural
  severity_at_least: high
then:
  - set_autonomy: approval_required
require_human_to_restore: true

The runaway-refund scenario ends differently with this armed: cost-per-run drifts critical at 02:00, autonomy tightens to approval_required, and the 03:00 refund parks in an inbox instead of executing. The rails are deliberate — a policy never loosens, suspension is opt-in with a second acknowledgement, one action per cooldown, and restore is always human (automatic restore flaps). Every action and every considered-but-not-fired decision is recorded, as actor system:drift-response, with the blast radius previewed before you arm.

Contracts and conformance — prove the permissions

The contract you minted at deploy keeps working after launch:

Config changes → create a new version (Contract → Active → New version from current config; the button is enabled only when the live config differs from the active version); the stored diff separates widened from narrowed, and the approver sees exactly the diff they approved, forever. Re-attesting an unchanged version is disabled for the admin already stamped on it — a second signature has to be a second person.
Conformance → because contracts are generated from live config, “is this system running configuration nobody approved?” is computable — and stated plainly on the Contract → Active sub-tab, next to the version history.
What the contract bounds and what backs it → Contract → Operating envelope holds the enforced per-run ceilings (tokens, cost, actions per task) that stop a runaway loop mid-run, and the graded bounds (rates over the reporting window) that the report marks met, breached or insufficient evidence but that never block traffic; Contract → External evidence records what humans produced outside the platform — an impact assessment, an evaluation or red-team report — where it lives, how to verify it, and when it expires, because the gate for a high-risk system asks for an unexpired one.
Declared vs observed → an undeclared client calling your system, a declared client that never calls, a granted resource never used (least-privilege finding): all reported once there’s enough traffic to mean something.

Before shipping a config change, replay the candidate contract against a stratified sample of real recorded runs — rarest-first sampling, so the suite includes the exhausted runs and rare tools a happy-path sample misses. outcome_changed (an action that used to succeed would now be denied) fails the gate; a new approval requirement is reported as policy_changed, stricter-not-broken.

Health — one number you can decompose

health = min(liveness, behaviour, reliability, cost, conformance, oversight, review)

Worst-of, never an average — a system that is cheap, fast and dead is not 80% healthy. Only healthy renders green; learning and unknown are amber because they mean unmeasured, and unmeasured is exactly what an optimistic dashboard rounds up. The chain-integrity percentage sits beside the score so you know what it’s worth — and so does the share of evidence that was reported through the OTel inlet rather than witnessed by the gateway.

The sixth signal, oversight, scores the human control loop rather than the machine: approvals that expired with nobody answering, and decisions concentrated on a single approver. It applies only to systems that use approval gates — a system with no gates scores null, never zero, because no gate configured means there was no oversight requirement to fail. That is a question about its contract, not a mark against it.

The seventh, review, scores whether the findings the platform raised were actually looked at: an open critical Inbox item older than the review SLA drags the verdict down for every system kind. Nothing about a system’s shape excuses nobody reading its findings.

The Assurance Report

The system’s Assurance tab opens on a diagram of the assurance loop — evidence, learned normal, signals, verdict, human review, report — with the system’s current position marked and live facts on each node; the findings open count is a button straight into the Inbox, because that node is the one only a human can advance. Four sub-tabs follow: Conclusion (the verdict, its basis and what it does not cover), Evidence (the worst-of health block with all seven components, the evidence-maturity ladder, and the learned baselines with their run counts), Exceptions & changes (attestations, profile relaxations, contract changes) and Snapshots (the captured reports). Export PDF and Export JSON sit in the tab header and export the live report.

The report itself is one artifact per system, answering every question in the table at the top with evidence: identity and declared intent, contract history with approvers and hashes, lifecycle transitions with who approved and who recorded them, drift events with causes and responses taken, run outcomes and cost per completed run, liveness record, replay results, conformance, and the human-review trail from the Inbox. The engineer opens it at 09:00; the auditor exports it in December — same artifact.

Snapshots are immutable (evidence that changes when the system does isn’t evidence), and every report ends with what it does not cover — chain integrity below 100%, the share of runs that were reported rather than witnessed (evidence sources), guardrails not re-evaluated in replay, clients that never traversed the gateway. What the platform keeps about a run, for how long, and what a data-subject erasure removes from the evidence is set out in Data handling. Captured reports feed ISO 42001 control coverage (A.6.2.4 performance evaluation, A.6.2.7 post-market monitoring) automatically.

curl http://localhost:5050/v1/admin/ai-systems/$G/assurance \
  -H "Authorization: Bearer $ADMIN_TOKEN"          # the live report
curl -X POST http://localhost:5050/v1/admin/ai-systems/$G/assurance \
  -H "Authorization: Bearer $ADMIN_TOKEN"          # capture an immutable snapshot

Tamper evidence — proving the records themselves

Everything above reasons over proxy_logs. That raises an obvious question an auditor will ask: how do you know those rows say what they said when they were written?

Every audit row carries a SHA-256 covering its own contents and its predecessor’s hash, so altering any row breaks every hash after it. The covered contents include a digest of the request and response bodies, sealed before retention truncates, redacts or prunes them — what a tool was asked and what it returned stays provable after the text itself is gone — and, for tool calls, the hash of the tool definition and the server version that were in force, the model that decided the call, and the calling agent’s declared release (agent implementation). The audit row lists every field, exactly what the hash covers and why the rest is outside, and publishes the byte encoding so exported rows can be verified without Brutor. Chains are per proxy process — a restart begins a new one, which is expected. Batches of rows are signed with Ed25519 into audit_chain_checkpoints, because hashes alone only detect edits: anyone able to rewrite the rows could recompute the chain. A signature is what makes that require a key.

Verify from System → Audit Chains in the Admin Console, or:

curl -X POST "http://localhost:5050/v1/admin/audit/chains/$CHAIN/verify" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Six verdicts, because “not verified” covers situations you would act on differently:

Verdict	What it claims
`verified`	Hashes link and a checkpoint signed by a trusted key covers them. The only verdict asserting integrity.
`unattested`	Hashes link; nothing signed them. No signing key configured — see `BRUTOR_AUDIT_SIGNING_KEY`.
`gap`	Sequence numbers missing: rows were deleted.
`broken`	A row no longer hashes to its stored value.
`forged`	A signature verifies against a key this deployment never trusted. Ranked worst — a hash break can be corruption, an unauthorised attestation is deliberate.
`unchained`	No chain columns. Rows written before tamper-evidence was enabled — not a pass and not a failure.

Consistent with the rest of assurance: absence of evidence is never reported as evidence of integrity, and a check that could not run returns an error rather than a clean result.

Scheduled audit exports to S3 carry the same property outward. Each export writes a <object>.manifest.json sidecar with a SHA-256 over the delivered bytes — so a recipient verifies with sha256sum alone — signed by the same key. The manifest also records chain_columns_included, because a field allowlist that strips row_hash silently makes an export unverifiable and the recipient cannot tell from the file.

The row chain proves integrity to someone who trusts your key. To prove it to someone who does not, turn on sealed action records: every verdict is signed as an RFC 9943 statement into a per-tenant Merkle log, the log’s tree heads — and each row chain’s checkpoint — are countersigned by independent witnesses, and evidence bundles verify with any COSE/SCITT tool. See the compliance evidence programme and tenant logs & witnesses.

End of the journey — and the loop back

Retirement is a gate too: active → deprecated requires naming a successor (or stating there is none), and deprecated → retired requires 30 quiet days. Nothing simply disappears with its history.

More often, assurance sends you backwards around the loop: a finding lands in the Inbox, becomes a governance change (Govern), is tested by replay, becomes a new contract version an approver signs, and is observed again (Observe) — one AI System, one thread of evidence, the whole way around.

How assurance worksThe concepts from first principles — the four guarantees and the full cycle, with a detail page per signal.

Assurance APIEvery endpoint referenced on this page.

Fleet viewZoom out: every AI System, asset and identity in the tenant, with governance gaps.