Behavioural drift

Brutor learns what normal looks like for each system from that system’s own history, then reports when the behaviour moves — with the likely cause attached. There is no universal “correct” number of actions per task; the only honest reference point is what this system used to do.

The statistics are deliberately unfashionable

Every measure is robust: median and MAD rather than mean and standard deviation, Jensen–Shannon divergence rather than KL, nearest-rank percentiles. Run metrics are heavy-tailed — one agent that loops 400 times moves a mean enough to hide a real shift in the other 500 runs, and a standard deviation computed over that tail is wide enough to swallow anything.

A change must also be sustained, not a single excursion: a metric has to exceed the robust-z threshold and have a located change point. One bad afternoon is not drift.

Learning is not health

Below 50 baseline-eligible runs a system is learning and emits no drift alerts at all. The UI says “still learning this system’s normal” rather than showing an empty list that reads as a clean bill of health.

The cause, not the p-value

The alert Brutor tries not to write is:

action_count up 180%, p=0.001

What ships instead is:

action_count up 180% since 14:00 on the 14th; the provider shipped a new point release of the underlying model at 13:52.

Brutor is the only component that sees both the traffic and the model registry, so it is the only thing that can make that correlation. Candidate causes are checked in order — model change, dependency/tool change, composition change, input drift, and so on — and the ladder terminates at unexplained, which is reported as such. An unexplained finding offers the correlated metrics as a starting point instead of inventing a reason.

Severity is consequence, not confidence

A statistically bulletproof 3% cost rise is low. A noisy 4× jump in max-iteration exhaustion on a refund system is critical. Severity comes from effect size × run volume × business impact (cost delta, autonomy level, EU AI Act risk tier). Ranking by confidence rather than consequence is how a drift feed becomes noise.

Triage

Each finding can be acknowledged (I am looking at this), resolved (it was real and is handled) or marked not drift (the detector was wrong). The last is deliberately a separate control: only it is evidence about the detector, and a detector that cannot be told it was wrong gets muted within a month.

Systems whose behaviour is legitimately erratic by design can be excluded entirely with baseline_opt_out — better an explicit switch than an operator muting the whole alert category.

The run ledger — where the baselines come from (and why partial runs are excluded)
Autonomy & response — acting on a drift finding automatically
The gate & replay — drift answers “did it change?” after the fact; replay answers “would this change it?” before