How the Gauge was tested

Volume, pageviews and editor intuition cannot tell you which of your output a subscriber could act on. A style guide says how sourcing should be done. It cannot show you how well it is done, sentence by sentence, across a title.

I wanted to build something that could: a way to distinguish reporting, research and commentary that changes a reader’s decisions from content that only occupies their attention.

The first version, in June 2025, was a rubric held against a gold standard for high-trust information businesses. It was built round three dimensions, which I now put as three questions. Signal: does it tell this reader anything? Confidence: is the claim grounded? Actionability: does it change what they would do?

The early versions flattered some of what they read. Signal and Confidence ran high, and Actionability was handed out too freely. So I wrote the standards into the software. Every piece is scored against the same standards in the same way, the model does not decide for itself what good looks like, and anything in its output that breaks them is flagged for review, never silently corrected. Then I read real output sentence by sentence against my own judgment, and rebuilt the scoring until the three dimensions stopped moving together.

Strong content does not score evenly.

A sentence on urbanisation, precise data explaining a non-obvious mechanism, scored Signal 8 and Actionability 3. “The content is not weak. It is explaining, building the causal architecture a reader needs before any decision becomes possible.”

A think-tank commentary ran 34 consecutive sentences without a Confidence score above 6. “This is not a sourcing failure but a signature of the genre.” In genres like that, Signal is what tells one piece from another.

Both from an earlier run.

In a run on 4,091 sentences from one specialist sector, Confidence varied as much between articles from the same publisher as it did between publishers. The inference: one initial use of the Gauge is to see how much a title varies from piece to piece, and whether that variation is intended. As AI produces more content that averages everything, what sets journalism and research apart (specific insight, sound grounding, consequence for the reader) is what is worth measuring.

One pass, the same instrument every time. Each piece is scored once, by the same model with the same set-up, and every run carries a record of what produced it. I tested whether a second pass would change the answer: against a written reader profile, two separate runs agreed on 91% of sentences about whether each bore on that reader’s decisions. Ten articles, 436 sentences. So the Gauge doesn’t average repeats. Each result is kept as it came, with its reason. Any figure can be walked back to the sentence it came from. A first run is 15–20 articles from one of your key titles; the picture builds from the pieces.

Experts agree. Where the Gauge said a sentence bore on a decision, an expert reading blind agreed four times in five.

Every score can be challenged. Each score comes with its reason and a record of the run that produced it, so it can be inspected, questioned, dismissed or re-run. You can see this on every sentence of one of my own pieces: See it on a real piece.

It scores pieces and titles, never writers, and measures content against a defined reader’s actions: a sentence that is news to a generalist can be old news to a specialist, so the reader is recorded before scoring starts.

It scores what is on the page, so it can miss a sentence whose importance depends on what an expert reads into it: a bare figure, an unstated mechanism.

Quality becomes something you can steer, align on internally, price against, and show a buyer.