A 48-hour hackathon build. Point it at a supply-chain advisory and it answers three questions in order: what's the threat, are we exposed, what did that save us.
A model reads the article and extracts every affected package and version. Those go straight into real exposure checks against our own environment, and the result lands on a dashboard next to the cost of having produced it.
483 out of 484
On one compromised-package advisory the article named 484 affected packages. The extraction found 483 — missing one that was listed identically to six siblings it caught. Not a bad result, and precision was perfect: nothing wrong was extracted. But a missed package is a package you've quietly told the business you aren't running.
The fix isn't a better prompt. Ask one model call to both grade every package it extracted and report what it missed, and it will reliably tell you it missed nothing — the bulk grading work crowds out the recall scan, every time. Ask the same model for recall alone and it finds the gap in about twenty output tokens.
Precision and recall are different questions, and a model can only really hold one of them at a time. Splitting them into separate calls was the whole improvement.
Recall also can't be chunked. Show the checker half the extracted list and every package outside that chunk looks missing — so verdicts batch freely, but the recall pass has to see the entire list in one call.
Getting the last one
It did get found, but not on the first ask. A single recall pass hits the tail maybe three times in five, so it runs a few rounds and unions whatever they name. That casts a wide net and drags in false positives — names mentioned in passing, attacker infrastructure, links to unrelated advisories.
The obvious filter is frequency: trust whatever shows up in most rounds. That's exactly wrong. The candidate appearing in four of five runs was a false positive, while the genuine miss showed up in only three.
So a second pass adjudicates every candidate against the source instead — is this named as an affected package, or just mentioned? Only confirmed ones survive, and they come back with their full version list, ready to scan.
Recovered packages get re-run through the exposure engine and merged back into the stored signal, so the dashboard shows all 484. Four model calls, three of them about twenty tokens each — and the confirm pass is skipped entirely when nothing surfaced.
Showing the value
The dashboard pairs two numbers per signal. GenAI spend is real, priced off actual token counts at published rates. Analyst time saved is an assumption, stated as one: two hours of manual triage per signal, valued at $50/hour fully loaded.
Two hours is a reasonable estimate rather than a measurement, and the code says so where the constant is defined. The ratio is the point — triage that costs cents in model spend against a task that costs an analyst most of a morning, on an article naming hundreds of packages nobody was going to check by hand anyway.
One Gemini detail worth keeping
Extraction runs with thinking switched off. Left on, the flash model spends its entire output budget reasoning and emits zero answer tokens — you get a truncation error and an empty result. Pulling a package list out of an article is transcription, not reasoning. No thinking needed, and no frontier model either.