Why AI systems are different
You have monitored software for years: health checks, error rates, dashboards, alerts. AI systems break that playbook — not because the tooling is bad, but because the failure modes are different in kind.
Four properties traditional IT never had
Section titled “Four properties traditional IT never had”- The engine is non-deterministic. The same input can produce different output on
every call. You cannot write
assertEqualsagainst an LLM, and a passing test yesterday proves little about today. - Cost is a runtime variable. Billing is per token, and how many tokens a task takes depends on what the model decided to do. A chattier loop triples spend with zero code changes.
- Your dependency updates under you. Model providers ship new point releases of the models you depend on — answers change tone, length and tool choices without a release note you deployed.
- The blast radius is real. An agent’s loop runs in your process with real side effects: emails, refunds, database writes. A misbehaving system doesn’t just return wrong text — it does wrong things.
Failures are quiet, not loud
Section titled “Failures are quiet, not loud”Thirty years of ops practice assumes failures are loud: exceptions, 500s, pages. The failure modes of AI systems are mostly quiet:
| Classic failure | AI-system failure | What your dashboard shows |
|---|---|---|
| Process crashes, throws, pages someone | Agent quietly starts taking 30 actions instead of 8 | Green — every call returned 200 |
| Dependency times out | Provider silently updates the model; answers change tone, length, tool choices | Green — latency unchanged |
| Disk fills, alert fires | Cost per task triples because the loop got chattier | Green — until the invoice |
| Service goes down, health check fails | Scheduled agent stops being invoked at all — zero errors, because zero traffic | Green — perfectly, absurdly green |
Request-driven metrics are functions of traffic that arrived. A system that degrades inside successful requests — or stops producing requests entirely — is invisible to all of them.
The consequence: a new operational discipline
Section titled “The consequence: a new operational discipline”Assurance is the discipline of making these quiet failures loud, without drowning operators in false alarms. Because there is no fixed “correct” behaviour to assert against, the reference point has to be each system’s own history — what normal looked like — plus what the system declared it would do at sign-off. That is exactly what Brutor builds:
- evidence of everything the system actually did — the run ledger
- a learned baseline and drift detection against it
- a dead-man’s switch for the silent failure — liveness
- a frozen record of what was approved — contracts
- one honest verdict, worst-of, never an average — health