The behaviour change benchmark
radius tested on 51 real release notes that changed behaviour. What it finds, and what it misses.
What is tested
A release note says something changed. Does radius find the lines of code it affects?
The benchmark checks this on 51 real cases. Each case is a real upgrade, like dotenv 16.6.1 to
17.0.0, with:
- the real release notes, copied exactly,
- a small project that uses the package like a real app,
- the lines the change affects, written down before radius runs.
A case is found when radius shows the note and points at one of those lines.
Only changes that still compile are tested: a new default, a stricter check, a different result. A removed function is caught by the type check, so it's not tested here.
The results
| Question | Answer |
|---|---|
| How many changes does radius find at the exact line? | 24 of 51 (47%) |
| How many does it wrongly call quiet? | 0 of 51 |
| Of the notes it shows, how many are the change being tested? | 38% |
The 51 cases were added in three rounds:
| Round | Cases | Found |
|---|---|---|
| First 14, written while building radius | 14 | 12 |
| Next 25, written without running radius on them first | 25 | 10 |
| Last 12, added later to check the fixes, same rule | 12 | 2 |
The first round was too easy: it was written by someone who knew how radius works. Rounds 2 and 3 are the fair score: 12 of 37, about one in three.
Quiet: from 13 mistakes to 0
Before, radius called 13 of these upgrades quiet. The code was affected, the notes said so, and radius said "nothing to read". It happened because the note didn't name anything radius could link to the code.
Now, if a note describes a change radius can't link to your code, the update can't be quiet. radius says review and lists the note.
The cost: quiet is much rarer. On 451 past upgrades of three open source projects (umami, uptime-kuma and outline), quiet went from 180 (40%) to 43 (10%).
On the first 39 cases:
| Version of radius | Found | Called quiet |
|---|---|---|
| When this page was first published | 18 of 39 | 8 |
| After three matching fixes | 22 of 39 | 5 |
| Now | 22 of 39 | 0 |
The three fixes:
- Without types, the options passed in the code count, like
limitinbodyParser.json({ limit }). - A note naming a nested option, like
retry.methods, matches the option. - A commit title that starts with a name, like
diff: fix ..., matches that name.
On the last 12 cases, these fixes found only 2.
Found
| Package | Upgrade | The change |
|---|---|---|
body-parser | 2.2.2 → 2.3.0 | An invalid limit now throws |
class-validator | 0.13.2 → 0.14.0 | forbidUnknownValues is enabled by default |
commander | 12.1.0 → 13.0.0 | Excess command-arguments are an error by default |
date-fns | 3.3.0 → 3.3.1 | getISOWeek rounds again instead of truncating |
dotenv | 16.6.1 → 17.0.0 | quiet defaults to false: config() logs at startup |
express | 4.21.2 → 5.0.0 | res.redirect('back') is no longer a magic string |
express | 4.21.2 → 5.0.0 | res.status() only accepts integers |
helmet | 7.2.0 → 8.0.0 | CSP directives like self must be quoted |
helmet | 7.2.0 → 8.0.0 | Strict-Transport-Security max-age is 365 days |
i18next | 22.5.1 → 23.0.0 | returnNull defaults to false |
js-yaml | 3.14.1 → 4.0.0 | dump() drops keys whose value is undefined |
jsonwebtoken | 8.5.1 → 9.0.0 | verify() rejects unsigned tokens by default |
ky | 1.14.1 → 1.14.2 | Uppercase retry.methods are retried |
lru-cache | 9.0.3 → 9.1.0 | set(key, undefined) deletes the key |
luxon | 3.2.1 → 3.3.0 | Interval#count no longer counts a midnight end day |
multer | 2.3.0 → 2.4.0 | Non-integer limits values are rejected |
pg | 7.18.2 → 8.0.0 | ssl verifies certificates by default |
react-router | 7.8.1 → 7.8.2 | generatePath() encodes param values |
semver | 7.6.3 → 7.7.0 | diff() from a prerelease to a stable version |
semver | 7.6.3 → 7.7.0 | inc() validates the prerelease identifier |
socket.io | 2.3.0 → 3.0.0 | Socket#join() no longer takes a callback |
validator | 13.12.0 → 13.15.0 | isEmail rejects addresses starting with a quote |
zod | 4.4.3 → 4.5.0 | z.iso.datetime() requires seconds |
zod | 4.4.3 → 4.5.0 | String length counts code points |
Missed, and why
In each miss, the note describes a real change that affects the project, but radius can't link the note to the code. There are four reasons.
1. The note only uses plain words
The note doesn't name anything the code uses. This is the most common reason.
| Package | Upgrade | The note says |
|---|---|---|
chokidar | 3.6.0 → 4.0.0 | "Remove glob support" |
csv-parse | 6.2.1 → 7.0.0 | "align trim with ECMAScript whitespace" |
jose | 5.10.0 → 6.0.0 | "private KeyObject instances can no longer be used for verify operations" |
knex | 3.1.0 → 3.2.0 | "Prevent unexpected combinations of statements and clauses groups from executing" |
marked | 7.0.5 → 8.0.0 | "deprecated options removed" (headerIds is ignored) |
marked | 14.1.4 → 15.0.0 | "escape html in renderers instead of tokenizers" |
papaparse | 5.3.2 → 5.4.0 | "Rename duplicate headers" |
sanitize-html | 2.10.0 → 2.11.0 | "Remove empty non-boolean attributes" |
vue-router | 5.0.6 → 5.0.7 | optional params are no longer set to an empty string |
yaml | 1.10.2 → 2.0.0 | "drop undefined values" when creating a mapping |
yup | 1.1.0 → 1.1.1 | "make null validation errors consistent across schema" |
zod | 4.4.3 → 4.5.0 | "improve french translation", in the commit list only |
2. radius can't see the option
The note names an option, but radius can't tell that the code's call uses it. For example, the option is defined in another package, or written as a link, or passed to a constructor.
| Package | Upgrade | The option |
|---|---|---|
@tanstack/react-query | 5.100.13 → 5.100.14 | "not subscribed", in words, declared in query-core |
ajv | 7.2.4 → 8.0.0 | strict, written as a link, not as code |
axios | 1.15.2 → 1.16.0 | maxContentLength on the fetch adapter |
fast-xml-parser | 4.0.15 → 4.1.0 | eNotation, an option of new XMLParser() |
ioredis | 4.28.5 → 5.0.0 | allowUsernameInURI, types from @types |
nodemailer | 8.0.11 → 9.0.0 | tls, types from @types |
qs | 6.14.0 → 6.14.1 | arrayLimit, types from @types |
react-hook-form | 7.61.1 → 7.62.0 | onBlur of register, from useForm() |
3. The note names something close, but not the same
The note names a related function, or writes the name in a way radius doesn't read yet.
| Package | Upgrade | How the note names it |
|---|---|---|
rxjs | 7.8.0 → 7.8.1 | throttle: and ThrottleConfig, for throttleTime |
sharp | 0.32.6 → 0.33.0 | tint as a bare code span, on a chained call |
zustand | 4.5.4 → 4.5.5 | setItem, called inside createJSONStorage |
4. The name belongs to something else
The note names something from another package, or a value inside a string.
| Package | Upgrade | The name |
|---|---|---|
body-parser | 1.20.3 → 2.0.0 | req.body, on Express's request |
passport | 0.5.3 → 0.6.0 | req#login(), on Express's request |
tailwind-merge | 3.6.0 → 3.7.0 | px and ps, class names inside strings |
ws | 7.5.10 → 8.0.0 | the 'message' event, a string |
What this doesn't tell you
- How often these changes happen to you. The 51 cases were picked by hand. They show whether radius links a change to the code, not how common changes are.
- How it does on big projects. The test projects are small. A big project uses a package in more places, so more notes match, including by accident.
- What happens on a miss. A miss still gets review. In 23 of the 27 misses, the note is listed among the notes radius couldn't link. In the other 4, other notes cause the review. What a miss loses is the exact line.
- Changes nobody wrote down. No tool that reads release notes can see them. Your tests can.
Where the cases are
The cases are in
packages/core/tests/benchmark.
pnpm benchmark prints the numbers on this page. They run offline, against a local copy of the
registry and GitHub.
Know a change radius should find? The README explains how to add a case. Or open a "wrong verdict" issue with the package, the two versions and the note.