Breaking good
Catching breaking API changes before your users do. October 2026.
A changelog is how developers find out about critical API changes. Fail to document a breaking change, and developers' integrations stop working. For a payments API, that means failed transactions and a very unhappy support queue.
Catching a breaking change often means manually comparing two versions of an API spec, change by change. It's slow, it's tedious, and it's easy to miss one.
To solve this issue, I asked Claude to build an Agent Skill that drafts the changelog, with breaking changes first. For those unfamiliar with AI skills, a skill is a folder of instructions that an AI coding agent, like Claude Code or Codex CLI, follows to carry out a series of specific tasks. I decided what the skill should do and how to test it, and I pushed back when I didn't like what was returned.
How it works
The skill doesn't let the AI decide what changed. It runs oasdiff, an open source tool that compares OpenAPI specs, and treats the output as the source of truth. The agent does the part a diff tool can't: it explains each change in plain language, tells developers what to do about it, and makes judgment calls.
Remove an optional field from a response, and oasdiff rates it as info, not breaking. Tell that to the dev who just had an API run fail. The skill I created lists it as breaking and tells me why.
When an answer can't be determined, the skill doesn't guess. The draft changelog includes a Needs review section with questions for the engineers managing the API. The list is the agenda for the review meeting.
How I tested it
I tested the skill on three API specs: a small AI generated one, and the public specs for Trustly North America and Payabli. A script made the same seven changes to each one, one change at a time, such as removing an endpoint or making a parameter required.
Claude Code and Codex CLI each wrote 21 changelogs. Then a different agent graded each one, so nothing graded its own work. Then I checked the graders.
What turned up
All 42 changelogs passed the automatic checks. The interesting part was everything else:
- A hallucination that wasn't. In early runs, Claude kept reporting that oasdiff rated a breaking change as "a warning, not an error." Three runs, same mistake. It looked like the AI making things up. It wasn't. oasdiff's Markdown output marks errors with a warning icon and leaves out the level, so Claude was reading the icon. I switched the skill to plain-text output, which prints the level, and the next three runs got it right.
- The graders needed grading. One AI grader passed everything it saw. Another got at least three things wrong that the spec disproved, and gave the same changelog different verdicts on different runs. An AI review is good for finding issues. It isn't always right.
- Stopping beats guessing. Codex stopped on 3 of its 24 runs, convinced that oasdiff wasn't installed. It was. Codex had misread its own check. The skill tells the agent to stop rather than guess, and it did. A rerun worked every time.
- It caught an unexpected mistake. To broaden the testing, I had Claude write an invented release with a dozen changes. The draft was complete and accurate, and it flagged a problem in the spec itself: a new endpoint for capturing payments that were "authorized earlier," in an API with no way to authorize a payment.
Where people fit
Before anything is published, a technical writer checks the draft against the spec, and an engineer who owns the spec answers the Needs review questions. I wrote that into the implementation instructions as a requirement, not a suggestion.
It doesn't need to be complicated. If your docs are stored in GitHub, the pull request launches the review, and merging the changes publishes the changelog.
What it doesn't prove
Every test used invented changes. I haven't run it on a real release with dozens of changes, so I can't tell you how much time it saves. At my next job, with the team's okay, I'll run it on a real release.
Try it
It's free and MIT licensed. Setup is copying one folder, installing oasdiff, and editing the style guide to match your team's changelog.
- The skill on GitHub, with setup steps for macOS, Linux, and Windows
- A sample changelog, with notes from reviewing the draft
- The test report, with every changelog and grade
Questions or feedback? Get in touch.