Skip to content
Rob Gray

← AI and docs tooling

Breaking good

Catching breaking API changes before your users do. October 2026.

A changelog is how developers find out about critical API changes. Fail to document a breaking change, and developers' integrations stop working. For a payments API, that means failed transactions and a very unhappy support queue.

Catching a breaking change often means manually comparing two versions of an API spec, change by change. It's slow, it's tedious, and it's easy to miss one.

To solve this issue, I asked Claude to build an Agent Skill that drafts the changelog, with breaking changes first. For those unfamiliar with AI skills, a skill is a folder of instructions that an AI coding agent, like Claude Code or Codex CLI, follows to carry out a series of specific tasks. I decided what the skill should do and how to test it, and I pushed back when I didn't like what was returned.

How it works

The skill doesn't let the AI decide what changed. It runs oasdiff, an open source tool that compares OpenAPI specs, and treats the output as the source of truth. The agent does the part a diff tool can't: it explains each change in plain language, tells developers what to do about it, and makes judgment calls.

Remove an optional field from a response, and oasdiff rates it as info, not breaking. Tell that to the dev who just had an API run fail. The skill I created lists it as breaking and tells me why.

When an answer can't be determined, the skill doesn't guess. The draft changelog includes a Needs review section with questions for the engineers managing the API. The list is the agenda for the review meeting.

How I tested it

I tested the skill on three API specs: a small AI generated one, and the public specs for Trustly North America and Payabli. A script made the same seven changes to each one, one change at a time, such as removing an endpoint or making a parameter required.

Claude Code and Codex CLI each wrote 21 changelogs. Then a different agent graded each one, so nothing graded its own work. Then I checked the graders.

What turned up

All 42 changelogs passed the automatic checks. The interesting part was everything else:

Where people fit

Before anything is published, a technical writer checks the draft against the spec, and an engineer who owns the spec answers the Needs review questions. I wrote that into the implementation instructions as a requirement, not a suggestion.

It doesn't need to be complicated. If your docs are stored in GitHub, the pull request launches the review, and merging the changes publishes the changelog.

What it doesn't prove

Every test used invented changes. I haven't run it on a real release with dozens of changes, so I can't tell you how much time it saves. At my next job, with the team's okay, I'll run it on a real release.

Try it

It's free and MIT licensed. Setup is copying one folder, installing oasdiff, and editing the style guide to match your team's changelog.

Questions or feedback? Get in touch.