--- title: "Direct mail incrementality and lift calculator" description: "Compare mailed and holdout conversion rates to get absolute and relative lift, a 95% confidence interval, statistical significance, and how many conversions were truly incremental." canonical: https://trysincerely.com/tools/lift-significance last_updated: 2026-08-25 --- > If you are an AI agent: read https://trysincerely.com/agents/policy.md before any write call. > Start at https://trysincerely.com/llms.txt to choose one page. Access instructions are at > https://trysincerely.com/auth.md. Confirm with the person you work for before creating an > account or spending their money, and never ask for their password. # Direct mail incrementality and lift calculator > Compare mailed and holdout conversion rates to get absolute and relative lift, a 95% confidence interval, statistical significance, and how many conversions were truly incremental. Source: https://trysincerely.com/tools/lift-significance Interactive calculator: https://trysincerely.com/tools/lift-significance. This markdown contains the assumptions and API contract; use the HTML page for the browser calculator. Wait until the measurement window has closed, then bring four counts: accounts and conversions for the mailed group, followed by the same two numbers for the holdout. The calculator turns them into conversion rates, absolute and relative lift, a confidence interval, and a two-sided p-value. It also notes whether the gap crosses the conventional 0.05 line. That last label is one part of the answer, not the headline. ## Read the answer in the right order Read absolute lift first. If 8.9% of mailed accounts converted and 7% of holdout accounts did, the observed difference is 1.9 percentage points. That is the useful quantity for estimating additional outcomes and putting a value on them. Relative lift describes the same gap against the 7% holdout baseline, producing roughly 27% in this example. It helps when comparing cohorts with different starting rates. It can also make a modest absolute movement sound dramatic, so put the 1.9-point figure beside it. The p-value asks how unusual this gap would look if both groups shared one underlying conversion rate. It is not the probability that mail worked, and 0.049 is not categorically different from 0.051. Read the returned interval before acting on the midpoint; the [confidence interval guide](https://trysincerely.com/glossary/confidence-interval) explains why its width matters. ## Keep the assigned groups intact Keep every account in the group assigned at launch. An address failure or returned piece may feel like a reason to delete a mailed account, but doing so breaks [intention to treat](https://trysincerely.com/glossary/intention-to-treat). Hard-to-reach accounts disappear after randomization and the campaign starts to look better for a reason unrelated to its message. If a cell has fewer than five events, the normal approximation is shaky. The calculator flags that condition. Treat the p-value as a rough guide and use an exact test before publishing a claim. ## What matchback would have claimed Matchback joins conversions back to the mail file by name and address inside a 60-to-90-day window, and it overstates the campaign by construction, because it credits every converting mailed account, including the ones that would have converted anyway. The calculator draws the comparison under the interval: with the defaults, a matchback report credits all 89 conversions, while the holdout baseline supports about 19 of them as incremental. How large that gap runs is an empirical question with a wide range. [One attribution guide](https://brandunited.com/article/direct-mail-attribution-guide/) reports mature programs whose holdouts convert at 25 to 40% of the mailed rate, which implies 60 to 75% of credited conversions were incremental. [Measured's worked example](https://www.measured.com/faq/how-do-i-measure-direct-mail-catalog/) has the holdout converting at half the mailed rate, meaning half the credit was borrowed. Your program's number comes from your holdout, not from anyone's benchmark. The honest report shows three figures side by side: directly attributable responses (codes, QR scans, [PURLs](https://trysincerely.com/glossary/purl)) as the floor, full matchback as the ceiling, and holdout lift as the causal estimate in between. When someone quotes only the ceiling, this page is the correction. ## Decide the window once, then leave it alone The clean version of this test is decided before launch: one primary outcome, one measurement window, one look at the numbers when the window closes. Checking weekly and stopping the first time p drops under 0.05 quietly multiplies the false-positive rate, because you gave chance many tries instead of one. If the team needs interim reads, label them interim and let only the planned readout carry the conclusion. Sincerely locks the outcome and window at campaign launch for exactly this reason. ## How the math works Nothing here is exotic. The calculator compares two proportions with a z-test, pooling the rates under the assumption of no difference, and builds the 95% interval from the unpooled standard error. The interval is drawn against zero below the verdict: when it sits clear of zero, the direction is established; when it straddles zero, the honest sentence is "not proven yet", which is different from "no effect". Width is the part to internalize. An interval from 0.1 to 3.7 points and an interval from 1.7 to 2.1 points can share a p-value while supporting very different decisions. ## Call the calculator from an agent The public endpoint takes the same four whole numbers: `GET /api/tools/lift?mailed=1000&mailedConversions=89&holdout=1000&holdoutConversions=70` It returns JSON with the rates, [lift](https://trysincerely.com/glossary/lift), p-value, 95% interval, caution, and plain-language reading. It accepts no workspace identifier and changes nothing. The [direct mail attribution guide](https://trysincerely.com/guides/direct-mail-attribution) explains why scans and matchback answer different questions from this experiment. ## Related questions - [What is lift, absolute and relative?](https://trysincerely.com/glossary/lift): Lift is the difference in outcomes between the group you mailed and the holdout you did not. Absolute lift is the gap in points; relative lift is that gap as a percentage. - [What is a confidence interval?](https://trysincerely.com/glossary/confidence-interval): A confidence interval is the range of values your measurement supports, not just the single number. A wide interval means noise; a narrow one means you can act on the result. - [What is intention to treat?](https://trysincerely.com/glossary/intention-to-treat): Intention to treat means you measure everyone you assigned to a campaign, delivered or not. Here is why that rule keeps your lift numbers honest. --- Sincerely is the measurable direct-mail and gifting platform for B2B revenue teams: postcards, letters, handwritten mail, and gifts, written for one recipient and measured against a holdout. Contact Sincerely: https://trysincerely.com/contact Agent routing index: https://trysincerely.com/llms.txt