Every recurring alert type has a runbook — a document explaining what the alert means, which systems to check, and what separates a real incident from noise. The first-line work is mostly mechanical: follow the doc, pull what it names, write up what you found. The judgement is a small slice of the effort; the rest is fetching.
This automates the fetching and the write-up, and leaves a report for a human. The runbook is the interface: onboarding a new alert type means writing Markdown, not code. A planning agent reads it and derives the plan the rest of the pipeline executes, so detection engineers extend the system in prose — in the artefact they already maintain.
The lesson: give the model less to do
Every real improvement came from taking structural responsibility away from the model.
Models decide content. Code decides sequence. Mixing those up was the source of most of what went wrong early on.
The first version had an LLM as the root router, picking which agent ran next. Replacing it with an explicit graph made control flow inspectable, testable and cheap — and killed off a whole failure mode where the router just quietly skips a step. Same story with the critic loop: "refine until the analysis is valid" became a hard iteration cap, because unbounded self-improvement is a cost incident waiting to happen and convergence past a few passes is rare anyway.
And it fails closed. Degraded paths emit an explicit undetermined verdict routed to a human, never a confident one. A broken lookup is not evidence of absence — conflating "nothing found" with "the query failed" is how an automated triage system silently closes a real incident.
Agent count is a cost, not a feature
It's easy to end up with seventeen agents because each one seemed reasonable when you added it. So I censused them: every agent in the tree, with a keep-or-collapse call, tested against a set of labelled real alerts. If removing an agent doesn't change a verdict, it earns its slot by being cheaper or clearer — not by existing. Several turned out to be orphans nothing had called in months.
That labelled set is also the only way to tell a refactor from a quiet accuracy loss. Its honest limitation: alert families missing from it are invisible to it, so a regression there wouldn't show up at all.