Skip to content
Rob Gray

← AI and docs tooling

Reviewing public docs with AI, and checking its work

Six companies, 12 pull requests, and two written reports. October 2026.

After my position at Trustly was eliminated, I reviewed the public documentation of six software companies. I wanted to show how I work in a docs-as-code workflow, and I wanted to find out how far an AI agent can take a documentation review before a writer has to step in.

Claude did the scanning and drafted the fixes. I set up the workflow, decided what counted as a problem, and checked every finding before it went anywhere.

How I ran it

Four of the companies have public docs repositories, so Claude worked from the source files. For the other two, I had Claude write a script that pulls each page's Markdown through the site's llms.txt index. Then codespell, a few Python checks, and Vale (Google style package, custom vocabulary) ran against that text. Vale wouldn't install in every environment, so it didn't run on all six.

I went one section at a time, and I had Claude read the pages, not just report what the linters flagged. That's how the more interesting problems turned up, like a code sample that contradicted its own comment.

For one 223-page docs set, I had Claude count terms across the whole site at once. Most of the consistency problems came from that pass. You won't see them reading page by page.

Before anything went public, I asked Claude to argue against its own findings and to tell me what it had actually verified versus what it was inferring. On one vCluster review comment, that meant hedging the single claim we couldn't prove. In the same review, Claude guessed wrong about where a file lived, and I corrected it.

I checked every finding against the live docs before submitting it. GitHub's Files changed view, or adding .diff to the URL, made that faster. If I couldn't confirm something, such as an issue that depended on how the HTML renders, it came out.

Then I applied the fixes in my own forks, opened the pull requests, and answered the maintainers' review comments. Whether to merge is their call.

What turned up

Plenty of typos and grammar problems, as you'd expect. The better finds were things a linter can't see:

Turns out technical writers still have a job to do.

What I didn't change, and why

Some findings didn't become edits on purpose. In GitBook's docs, pages disagreed about how permissions resolve, how Translations is sold, and a password deadline. Only the product team knows which version is right, so I listed those for the maintainers instead of guessing. Some GitBook pages are also generated from another repository, so editing them directly would have been the wrong fix.

In vCluster's docs, a table row for --repository-url looked copied from the row below it, with the wrong description and default. I couldn't confirm the correct values, so I left it alone.

And not every flag was real. After one fix, Vale raised a new warning on a command-line flag that was most likely a parsing quirk.

The result

Ten of the 12 pull requests have been merged: seven at Medusa and three at vCluster. GitBook's is under review, and one went to a repository that has since gone private. I also filed an issue at Medusa for a build problem on Windows. One of the two teams that received a written report shared it internally.

If I were turning this into something repeatable, I'd start with the cross-page term check. A per-page review can't catch those problems, and it's easy to run on every release.

Selected pull requests