A few months ago I was looking at a dashboard of quality scores at Frontward where almost every one was a 90-something. When a model hands you great scores, there’s something wrong.
One real score for a Happy Being ad. The ad is a creator unboxing a specific blueberry white tea. The landing page opens on inflammation science.
Parity grading checks when an ad makes a promise, does the page it points at keep that promise. The ad here is a creator unboxing a blueberry tea. The page opens on “Supporting a healthy inflammatory response.” Nobody is sitting there clicking every ad to feel that mismatch, so the money just quietly underperforms.
The 75 on top of that card almost buried the only row that mattered. Message: 1, meaning absent or contradictory. Offer and CTA are both 4s, Product and Proof are 3s, and none of that is the story. The page pivoted to a different subject than the ad that earned the click.
The same failure written as a fix: the creative leads with blueberry tea, the hero leads with inflammation, so rework the page to mirror the hook above the fold.
The total was the least useful part. Ask a model to rate a page from 0 to 100 and it hugs the top, and one of ours parked nearly everything between 96 and 100. A model from a different provider squashed the same pages down around 75 on a calibration rule we never wrote. Same pages, two unrelated distributions, and neither number told anyone what to fix. Breaking it into five dimensions on a 0-to-4 scale worked because models are bad at the global question and fine at small local ones, as long as each rung has a written definition.
Those definitions live in a markdown file loaded word for word into every scoring prompt. When the scorer is wrong you edit prose and ship a new version, which is why the version number on that card is most of the project’s history. v1.1 produced the wall of 96s. v1.2 wrote the specific failures down as rules: a generic “find your flavor” page caps at 3 when the ad pushed one product, and Message gets judged against what the video actually opens on rather than word overlap.
v1.3 barely touched the definitions and changed the posture instead. A 4 is exact parity above the fold, not close enough. A 3 is a real gap the client should fix. On a borderline call, take the lower score. The burden of proof is on the page. Telling the model to be skeptical rather than charitable moved the scores more than most of the code around it.
Models also kept breaking an invariant. A page only counts as a clean match if all five dimensions are 4s, but one will happily score something 4/3/4/4/4 and stamp “clean match” on it anyway, because the page felt clean. So a function recomputes that label from the sub-scores at save time. The first time we ran it over the existing database, eleven scores were labeled clean while carrying a sub-4 dimension.
The harder question is whether the grader is any good, and there’s no answer key. Correct parity is a judgment call, which is the whole reason this exists. What worked was scoring every page twice with two different models and looking only at where they disagreed. Small gaps get ignored, big ones get flagged and typed, so the flag tells you what kind of mistake to expect. A smoke test over six pages flagged five of them, not because five scores were wrong, but because those were the borderline calls where the rubric has a soft spot.
The loop only reports and never auto-corrects. When the thing being evaluated is judgment, an automated fix to the judge is a way to launder a bug into a fact. Underneath it sits a small set of pages two people graded by hand, and a new rubric has to land close to them before it ships.
I expected the hard part to be prompting, and modern models handled the shape of the task on the first try, including the message pivot I’d spent weeks failing to define. The work was deciding what a good answer was and writing it down clearly enough that two models grading separately mostly land in the same place.