A few months ago I was looking at a dashboard of quality scores at Frontward and almost every one was a 90-something. Most people would see that and feel good. I felt the opposite. When a model hands you a wall of high marks, the first thing to suspect is the model, not the work. So I grabbed one score and refused to let it go.
Here it is.
One real score for a Happy Being ad. The ad is a creator unboxing a specific blueberry white tea. The landing page opens on inflammation science. The model gave it a 75.
A 75 out of 100. Reads like a mediocre B-minus, the kind of number you skim past. Then look at the row that says Message: 1. The ad is a creator unboxing a specific blueberry tea. The page it points at opens on “Supporting a healthy inflammatory response.” The ad sold tea, the page gave a lecture about inflammation, and that single 1 is the whole story. The 75 on top of it nearly buried the one thing worth knowing.
Let me back up, because none of that means much until you know what we’re grading.
So what is parity
When a brand runs an ad, the ad makes a promise. “50% off your first box.” “I just got my hands on this.” A picture of one specific blueberry tea. You click, you land on a page, and the question that quietly decides whether the ad money worked is dumb and simple: did the page keep the promise the ad just made?
The industry calls this message match, or scent. We call our version parity, and it asks only whether this page paid off this ad. Not whether the page is pretty, or fast, or well written. A gorgeous page can score terribly, because the ad sold blueberry and the page opened on inflammation. The leak is invisible in normal reporting. The ad looks fine, the page looks fine, the money just underperforms, and nobody is sitting there clicking every ad to feel the mismatch. So we built a thing that does, and the card above is one of its outputs. The rest of this post is me taking that card apart, from the useless number on top down to the part that’s actually the product.
The number is the boring part
That 75 is a weighted sum. Five things an ad can promise, each graded 0 to 4, each worth a fixed slice of the total:
parity = Offer×6.25 + Product×6.25 + Message×5 + Proof×3.75 + CTA×3.75
Five dimensions, max 4 each, weights chosen so a perfect card is exactly 100. It has the shape of rigor. And on its own it tells you almost nothing, for a reason that’s annoyingly simple: models are bad at the global 0-to-100 question and decent at small local ones.
Ask a strong model “rate this page from 0 to 100” and it hugs the top. Early on, one of our models parked basically everything between 96 and 100. A different model from a different provider squashed the same pages down around 75 and made up a calibration rule we never wrote. Same pages, two models, two unrelated distributions, and neither number was something I could act on. A 97 doesn’t tell you what to fix. A lonely 75 doesn’t either. You’re looking at one right now, and by itself it’s noise. The breakdown underneath it is where the information actually lives.
So the first real decision was to stop trusting the total and start trusting the parts.
Cracking open the 75
Each of those five sub-scores is graded on a 0-to-4 scale, and the scale is the whole trick. It isn’t a feeling. Each rung has a definition:
| Score | What it means |
|---|---|
| 0 | Technically broken for this dimension (404, the section won’t load) |
| 1 | Absent or contradictory on a working page |
| 2 | Related, but materially different from the promise |
| 3 | Directionally right, with a real and fixable gap |
| 4 | Exact above-the-fold parity |
Read the card again with that in hand. Offer is a 4, the page carries the same “try it, free shipping” energy the ad had. CTA is a 4, the buy button is right there above the fold. Product is a 3 and not a 4, because the ad showed a specific blueberry 4-pack and the page hero is the generic lineup, so the visitor still has to go hunt for the thing they clicked on. Proof is a 3, because the page leans on a doctor badge and a wall of reviews instead of the specific proof the ad implied.
Those are the supporting cast. The lead is Message, and Message is a 1.
The 1 is the whole ad
A 1 means absent or contradictory, and here’s how the model justified it, in its own words on the card: the ad sells a blueberry tea unboxing moment, while the page opens on “Supporting a healthy inflammatory response” and “More happy days. Less inflammation.” The page pivoted. It walked the visitor in the door talking about a different thing than the ad that earned the click.
The same failure, written as a fix. This is the output that earns its keep: not ‘Message: 1’ but the sentence saying the creative leads with blueberry tea while the hero leads with inflammation, so the page should be reworked to mirror the hook above the fold.
This is the exact failure I’d been trying to describe in the abstract for weeks, and the model caught it cold on a real ad. It’s worth being precise about why it’s hard, because keyword matching gets it wrong every time. Both the ad and the page are “about” Happy Being tea. Both mention taste somewhere, both mention health somewhere. A naive overlap check sees the same words on both sides and calls it a match. But what the visitor sees first is what sets the expectation, and the first thing here is a mismatch: the ad opened on tea and a face, the page opened on a clinical claim. That gap is the message pivot, and it isn’t cheap. Dropping Message from a 4 to a 1 strips off as many points as losing the entire CTA would, which is part of why Message carries more weight per point than CTA in that formula above.
The 75 told me none of this. The 1, and the sentence under it, told me all of it.
Who graded this, and why I only half-trust them
A model produced that card. Which is the catch: a model produced that card. Models are generous, inconsistent, and easily charmed by a page that looks nice. If the grader is an LLM, the grader has to be governed.
The thing doing the governing isn’t code. It’s the rubric, and the rubric is a markdown file that gets loaded, word for word, into every scoring prompt. When the scorer is wrong, you don’t patch a function. You edit prose and ship a new version. The card says rubric v1.3, and that version number is most of the project’s actual history.
v1.1 was the naive rubric, the one that produced the wall of 96-to-100. v1.2 is where the specific failures got written down as rules. A single ad-relevant product has to be the hero to earn a 4 on Product, so a generic “find your flavor” page caps at 3 when the ad pushed a specific one. Message gets judged against the ad’s dominant creative claim, the thing the video actually opens on, not against word overlap. A generic five-star rating is not the specific proof an ad made. And one line that did a lot of work: don’t anchor to 3, don’t rail to 4, a perfect 100 should be rare.
v1.3 barely touched the definitions. It changed the posture. A few lines, roughly:
A 4 is exact above-the-fold parity, not “close enough.” A 3 is not a pass, it’s a real gap the client should fix. On a borderline call, take the lower score. The burden of proof is on the page.
We told the model to be skeptical of the page instead of charitable toward it, and the scores got more honest immediately. Those few sentences of attitude mattered more than most of the code around them. Not an algorithm, a tone of voice to grade in.
There’s also a quiet invariant the models kept breaking. A page is only a clean match if all five dimensions are 4. Obvious. But a model will score something 4/3/4/4/4 and still stamp “clean-match” on top because the page felt clean. So we never trust that label as written, a function recomputes it from the actual sub-scores at save time, and if the vibe disagrees with the data the data wins. The first time we ran that check across scores already in the database, eleven of them were labeled clean while carrying a sub-4 dimension. Eleven pages the system had quietly blessed that it should have flagged.
Two models, one page, the gap between them
The rubric makes one model more honest. It does nothing about the deeper question: is the grader any good? There’s no answer key. “Correct parity” is a judgment call, and the whole reason this exists is that humans don’t have time to make that call thousands of times.
The move that cracked it open was almost embarrassingly simple. Score every page twice, with two different models, and go look at where they disagree.
flowchart LR
P["Score in the DB<br/>(model A)"] --> D{"Diff vs<br/>critic"}
L["Same page,<br/>re-scored by<br/>critic (model B)"] --> D
D -->|"weighted gap ≥ 10<br/>or any dimension ≥ 2"| F["Flag it"]
D -->|"within tolerance"| OK["Agree, move on"]
F --> C["Classify the flag"]
C --> H["Human reads<br/>the short list"]
H --> R["Patch the rubric<br/>or fix the page"]
R -.->|"new rubric version"| PA score sits in the database, written by one model. The loop pulls it back out, re-scores the same page with a second model acting as a critic, and diffs them dimension by dimension. Small gaps it ignores. Big ones it flags, and the flags are typed, so the type tells you what kind of mistake you’re probably staring at: one model called it a clean match and the other didn’t, or one read the hero as a product match and the other didn’t, or one missed a message pivot the other caught. Those last two are the blueberry-tea failure from above, surfaced automatically before anyone had to click through the account.
The first time I ran this over real scores it was a smoke test, six pages, and it flagged five of them as worth a second look. Not because five scores were wrong. Because those were the borderline calls, the ones where the rubric has a soft spot, and disagreement is how they announce themselves. Two models that both say 96 give you a page you can usually push down the queue. One that says 91 next to one that says 78 is a page worth a human minute. The gap is the signal, more than either score is.
One choice I’d defend hard: the loop only reports. It never auto-corrects. Its job is to decide whether a rubric change is worth making and to point at the cases that justify it. When the thing you’re evaluating is judgment, an automated fix to the judge is just a way to launder a bug into a fact.
How I checked the grader
This is the part that stays uncomfortable, and I’ve made my peace with it. There is no dataset of correct parity scores, and there can’t really be one, because reasonable people will argue about a 3 versus a 4 all day. So instead of one accuracy number I lean on a few smaller checks that are harder to fool.
flowchart TD A["Calibration fixtures<br/>two humans grade ~10 pages by hand"] -->|"do the humans agree?"| B["Critic audit loop<br/>two models re-score, gaps surface"] A -->|"no, rubric is too vague"| A2["Rewrite the rubric, bump the version"] B --> C["Batch eval<br/>model vs fixtures: agreement + error"] C -->|"scores drift"| A2
The base layer is a small set of pages graded by two people by hand. That’s the bar a new rubric has to clear before it ships: the two humans have to agree closely, within a point most of the time, dead-on for the heaviest dimensions. If they can’t, the rubric is too vague to hand a model, and the fix is to rewrite the prose, not to nudge a threshold. On top of that sits the critic loop, two models arguing in production. On top of that, a batch eval that runs the model against the locked human-graded fixtures and reports how often it matches, how often it’s within a point, and the average miss on the total. When v1.3 made the model stricter, this is where I checked that “stricter” meant “closer to the humans” and not “differently wrong.”
This still isn’t proof. It just makes a bad rubric edit much harder to ship unnoticed.
A couple of smaller things that earned their keep. The model never re-watches the ad video. We extract the creative once, the hook, the offer, the on-screen text, and hand that summary to the scorer as evidence, with strict instructions to reason only from what it’s given. That’s the “scored with partial context” tag on the card. And validation is paranoid on purpose: a sub-score of 0 means the page is broken, so if the model emits a 0 without naming the failure, the result gets thrown out instead of stored. A confident, clean-looking, internally contradictory score is the most dangerous thing a grader can produce.
What the 75 actually bought
I went in expecting the hard part to be prompting. Getting a model to read a page, follow a rubric, emit clean JSON. That turned out to be the easy part. Modern models can usually follow the shape of the task on the first try, including, as it happens, catching a message pivot I’d struggled to define.
The hard part was everything wrapped around the model. Deciding the score was five judgments and not one. Writing down, in plain prose, the difference between a page that’s broken and a page that’s merely generic. Teaching the grader to distrust a pretty page. And, because there was no answer key, standing up a second grader whose only job is to argue with the first one so a human can see where the disagreement lives.
That blueberry ad is still live. Its card still reads Message: 1, with a concrete fix attached: open the hero on the tea, not the inflammation. The next time someone reviews this account it sits near the top of the queue, one specific claim a person can accept or reject in a minute instead of clicking through fifty ads to find it. That’s the version of the system I trust: a short review queue, not a score that pretends to settle the question. And the part that took the work was never the model. It was deciding what a good answer was, and writing that down clearly enough that two models grading on their own mostly land in the same place.