I hand-graded my own tool on 5 issues. Here are the 4 misses.

Someone on X asked how often escalation-gate catches stuff. Two or three good runs don't answer that. So I ran an eval on HackerRank's open-source hiring-agent: a fixed set of real issues, an answer key written by hand beforethe tool ran, and scoring rules fixed before any result was seen. Picks were made by a fixed rule, before reading issue titles, so I couldn't choose easy ones.

I stopped at 5 clean hand-graded cases: issues #131, #271, #273, #378 and #388. That is a small sample, and every number below should be read that way.

One caveat on my own key: for #388 I first said both PRs fix it, and moved #389 to “partly” after running the parser. So that answer was informed by running code, which the tool can't do yet.

The pattern matters more than the count. Every miss is a side effect or an input nobody tried. That follows from how v1 works: it finds the cause and checks every citation against the code, but it judges PRs by reading their diffs. It never runs them.

What v2 changes

First, a side-effect pass after every verdict: what else does this PR change? Removed names, deleted data, changed defaults. Then sandboxed runs: the tool writes a test that reproduces the issue, the test must failon the base commit (or the repro is fake), and must pass with the PR applied. The repo's own tests run before and after, so a PR that breaks something else shows up as a pass → fail. The four cases above are the regression set v2 has to catch.

Source: escalation-gate on GitHub · gate.outbuiltit.com