← Tools I Use

Anthropic open-sourced its vulnerability-hunting harness — the interesting part is the verifier, not the finder

2026-06-04 · ~8 min read · Conor Dobbs

Anthropic just published a reference implementation for finding and fixing software vulnerabilities with Claude. It hit the front page of Hacker News this week under the headline "open-source framework for AI-powered vulnerability discovery," and the repo — anthropics/defending-code-reference-harness — ships a set of skills for threat modeling, scanning, triage, and patching, plus an autonomous scanning harness you can point at a target and a /customize skill to retune it for your own codebase.

The easy story to write is "AI now finds zero-days." That is true, and it is not new — the more honest read of this release is about the boring engineering around the model, not the model itself. The part worth copying is how the harness decides which of its own findings to believe.

What the harness actually does

The pipeline is six stages, and none of them are "ask the model if there are bugs." It starts with threat modeling — you define what counts as a vulnerability for this target before anything scans, so the run has a notion of in-scope versus noise. Then a sandbox: the reference setup runs agents and targets inside gVisor-isolated containers with egress locked to the model API only, built from a pinned Dockerfile, so an agent that gets a target to execute something cannot phone home or wander off the box. Discovery partitions the target into focus areas and fans out parallel agents to scan them. Verification takes each candidate finding to an independent agent that tries to confirm it is actually exploitable. Triage deduplicates, assigns severity, and re-ranks. Patching generates a candidate diff with an independent reviewer and hunts for variants of the same bug.

Read that list again and notice how much of it is plumbing whose only job is to distrust the previous step. The model finds things in the discovery stage. Almost everything after that exists to throw most of those things away.

The false-positive problem is the whole game

Anyone who has run a static analyzer in anger knows the failure mode of automated security tooling, and it is not "it misses bugs." It is that it cries wolf. A scanner that reports two hundred "critical" findings, of which six are real, is worse than no scanner, because triage fatigue means a human stops reading at finding number thirty and the six real ones rot in the queue with the noise. The thing that makes an automated finder useful is not recall. It is precision you can trust enough to act on without re-deriving every result by hand.

The numbers Anthropic published around this work are interesting precisely on that axis. In their write-up, one team's model findings were exploitable about 90% of the time when the threat model was well documented. Adding an adversarial verifier — a second agent whose job is to try to confirm or refute the finding — halved the false-positive rate coming out of discovery. Requiring an actual proof-of-concept before a finding is reported drove false positives to near zero, and one pentesting team with tooling to query the running application reached close to a 100% true-positive rate. As of late May the company reported it had disclosed 1,596 vulnerabilities and seen 97 patched off this line of work.

The lever in every one of those numbers is verification, not a smarter finder. The model that discovers the bug and the model that confirms the bug are the same family of model. What changes the output from "interesting research demo" to "a security team will actually run this" is the architecture around the call: an independent skeptic, and a hard requirement to demonstrate the bug rather than assert it.

Why this generalizes past security

I keep harping on this because it is the single most transferable idea in the whole release, and it applies far beyond finding CVEs. Any time you put an agent to work on something where being confidently wrong is expensive — security findings, code review, data extraction, financial reconciliation — the same pattern shows up: a single pass that generates plausible output is cheap and untrustworthy, and the value comes from a second, independent pass whose only job is to refute the first. Generation is easy. Verification is the product.

That is how we build the things we run. When an agent here drafts a finding or a change, it does not get to mark its own homework; a separate pass with a different prompt tries to knock it down, and a claim that cannot survive that round gets dropped, not shipped. Anthropic's harness is the same instinct, formalized and pointed at vulnerability discovery, with a sandbox and a PoC requirement standing in for "prove it." If you are wiring agents into any workflow where wrong answers cost real money, the harness is worth reading less as a security tool and more as a worked example of how to make an agent's output trustworthy.

How it fits the supply-chain story we have been telling

This release lands in the middle of a month we have spent writing about the code you run that you did not write — the self-propagating npm worm, auth tokens leaking through published packages, and a VS Code extension that lifts your GitHub token in one click. The through-line is that modern software is assembled, not authored, and the attack surface lives in the seams. Automated discovery cuts the other way on the same problem: if attackers can fan agents across your dependencies looking for the weak seam, so can you, and the cost of doing it continuously is collapsing.

That is the genuinely important shift. Vulnerability discovery used to be a scarce, expensive, human-gated event — a pentest you booked once a year. A harness that runs recon, find, verify, and report on a schedule turns it into something closer to a cron job. The defenders who win the next year are the ones who treat scanning their own code and their own dependencies as continuous infrastructure rather than an annual ritual, and who have the verification discipline to act on the output instead of drowning in it.

The catch worth saying out loud

None of this is free, and the honest caveat is the sandbox. The reference harness isolates agents in gVisor containers with egress locked down for a reason: you are running an autonomous agent against code, asking it to find ways to break that code, and handing it a sandbox in which to try. That is a capability you want pointed at your own targets under your own control, not something to fire blindly. The same machinery that finds your bugs finds everyone's, which is exactly why Anthropic ships it with the isolation baked in and gates the hosted version. Run the demo on a known-vulnerable library first, read what the verifier rejects as carefully as what it accepts, and treat the patches it proposes as candidates a human still signs off on — the patching stage generates diffs, it does not get to merge them.

The takeaway that outlasts the repo

Clone it if you ship code, run the quickstart against the demo target, and watch what the verification stage throws away — that is where the education is. But the lesson that survives this particular release is architectural, not about security at all: the way you make an autonomous agent trustworthy is to never let it be its own judge. Generate with one pass, refute with an independent one, and demand a demonstration before you believe a result. Anthropic just shipped a clean, public example of that pattern doing real work. The bugs it finds are the headline. The verifier that decides which ones are real is the part you should steal.

Sources