Program Ledger

// why keep a ledger?

The most dangerous question in CRO: "you announced +40% cumulative lift this year, so why is revenue flat?" Every win gets celebrated on the day it ships and never checked again. The claims compound in the slide decks while the revenue curve does whatever it was already doing. Individually, each win looked real. Collectively, they describe a company that should be twice its size.

The companies that run experimentation at scale know this gap well. Microsoft and Google have both published that only 10 to 20% of their experiments produce genuinely positive results, and teams that measure program-level impact with long-term holdouts consistently find that shipped wins deliver a fraction of what was claimed. Novelty effects fade, inflated estimates deflate, and tests interact and partially cancel each other out. None of that is visible if nobody ever adds it up.

The ledger adds it up. Log every shipped winner with its claimed and honest lift, enter your actual monthly numbers, and see the three lines on one chart: what was promised, what was plausible, and what happened. No platform builds this view, because it audits them too. A testing program that never reconciles its claims with reality drifts into fiction, one winner at a time.

"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."

ATTRIBUTED TO MARK TWAIN

01 — Shipped Winners

Test	Shipped	Claimed	Honest

No wins logged yet. Every test you ship as a "winner" goes here — with both the number you announced and the honest one.

02 — Monthly Actuals

Month	Visitors	Conversions	CVR

No months logged yet. Enter at least your baseline month (before the first shipped win) and the months after — same source, same definition, every month.

03 — Claimed vs Reality

Log at least one shipped win and two months of actuals to see the confrontation.

04 — The Reading

—

Realization Rate

—

of claimed lift showed up

Waiting for data.

Program Stats

—

Honest caveats

Monthly CVR moves for many reasons besides your tests: seasonality, traffic mix, pricing, campaigns. A gap between claimed and realized is not proof your wins were fake, and a match is not proof they were real. But a persistent, growing gap across many tests is a pattern. Industry-wide, realization rates of 10 to 30% are common for programs that report naive (unshrunk) lifts. That's the winner's curse compounding.

Backup

// the math — how the confrontation is computed

Baseline

Your first logged month sets the reference point: baseline CVR = conversions / visitors of that month. Everything else is measured as change from there, which is why the baseline month should come from before your first shipped win.

The expected CVR paths — compounding claimed lifts

If every announced lift were real and permanent, each shipped win would multiply CVR from its ship month onward. For any month m:

expected(m) = baseline × Π ( 1 + lift_w /100 ) over all wins w shipped in or before month m

The chart draws this twice: once with the claimed (observed) lifts — the grey line — and once with the honest (shrunk) lifts from Reality Check — the blue line. Multiplication rather than addition because lifts compound: two +10% wins claim 1.1 × 1.1 = +21%, not +20%.

Actual path

actual(m) = conversions(m) / visitors(m)

Straight from your analytics, no modelling. Consistency matters more than precision here: same source, same conversion definition, every month, or the comparison means nothing.

Realization rate

Comparing total growth since baseline in the latest month:

realization = ( actual/baseline − 1 ) / ( claimed_expected/baseline − 1 )

100% means reality delivered everything the claims promised. The tool also computes realization against the honest line — when that one is near 100% while the claimed one is low, the diagnosis is clear: the program works, the announced numbers were inflated.

What this deliberately does not do

No seasonality adjustment, no traffic-mix correction, no causal attribution. Monthly CVR moves for many reasons, so a single month proves nothing in either direction. The signal is the trend: a gap that persists and grows across many months and many tests. Treat the realization rate as an honest conversation starter, not a verdict — that's also why average claim inflation (claimed ÷ honest per win) is shown alongside it.