---
title: "Direct mail holdout test example, from setup to readout"
description: "A fictional B2B direct mail holdout test worked end to end: power check, arm sizes, outcome counts, lift with its interval, and an inconclusive variant."
canonical: https://trysincerely.com/guides/direct-mail-holdout-test-example
last_updated: 2026-10-11
---
> If you are an AI agent: read https://trysincerely.com/agents/policy.md before any write call.
> Start at https://trysincerely.com/llms.txt to choose one page. Access instructions are at
> https://trysincerely.com/auth.md. Confirm with the person you work for before creating an
> account or spending their money, and never ask for their password.
# Direct mail holdout test example, from setup to readout

> A fictional B2B direct mail holdout test worked end to end: power check, arm sizes, outcome counts, lift with its interval, and an inconclusive variant.

Source: https://trysincerely.com/guides/direct-mail-holdout-test-example

Quarry Lane Software is fictional, and so is every count on this page. Sincerely has no customer case study to publish yet and will not invent one, so this page works a single direct mail holdout test from the first decision to the final number instead. The arithmetic is real. Every figure comes from the functions behind the [holdout size calculator](https://trysincerely.com/tools/holdout-size) and the [lift calculator](https://trysincerely.com/tools/lift-significance), and the quoted report lines are what Sincerely prints for the same counts. Of 5,200 accounts, 3,867 were mailed and opened 317 opportunities, while 1,333 were held out and opened 79. That is a lift of 2.3 percentage points, with a 95% range of 0.7 to 3.8.

![Lift ranges in percentage points: US, 5,200 fleets, 0.74 to 3.81, ready to use; Canada, 1,400 fleets, minus 0.61 to 5.19, still learning; Canada, four cohorts, 0.09 to 3.08.](https://trysincerely.com/images/guides/direct-mail-holdout-test-example.webp)

*In this direct mail holdout test, the US and first Canadian results share a 2.3-point midpoint, but only the US range clears zero until four cohorts are pooled.*

## The setup

Quarry Lane sells maintenance software to regional trucking fleets. Its marketing lead, Dana, wants to know whether a printed letter to each fleet's head of maintenance opens sales opportunities that would not have opened anyway. Sales keeps its usual emails and calls for every account, mailed or not, so the only difference between the two groups is the letter.

Before anything prints, she writes down the contract:

| Decision          | Choice                                                                                   |
| ----------------- | ---------------------------------------------------------------------------------------- |
| Eligible accounts | 5,200 US fleets that fit the target profile                                              |
| Assignment unit   | The account, so every contact at one fleet lands on the same side                        |
| Primary outcome   | Opportunity created, read from the CRM                                                   |
| Window            | 90 days per account                                                                      |
| Baseline          | 6%: the share of comparable accounts that opened an opportunity in any 90 days last year |
| Holdout           | 25% of accounts                                                                          |

Sincerely locks the outcome, window, baseline and holdout share at launch, along with the assignment salt and the readout mode. A database trigger rejects any later change to them. Dana computed the 6% herself from a CRM export, so Sincerely records the baseline as stated, not measured. Not every Sincerely campaign has a holdout. Meeting follow-up, deal acceleration, signal-triggered and other descriptive campaigns report activity without claiming lift. So does any campaign judged on QR scans or landing-page actions, since a held-out account never gets a piece to scan.

## The power check

The first question is the smallest lift worth paying for, not the size of the list. Quarry Lane's finance model says the letters only pay for themselves if they raise the opportunity rate by about 40% relative, from 6% to 8.4%. A smaller effect would not change the budget, so the test does not need to detect one.

The holdout size calculator, given a 6% baseline, a 40% lift and 80% power, asks for 1,820 accounts per group in an even split. Nobody on the sales team will hold back half the list, so Dana tries other shares:

| Holdout share | Mailed | Held out | Accounts needed |
| ------------- | ------ | -------- | --------------- |
| 50%           | 1,820  | 1,820    | 3,640           |
| 25%           | 3,746  | 1,249    | 4,995           |
| 10%           | 9,508  | 1,057    | 10,565          |

A 10% holdout would need 10,565 accounts, about twice what she has. A 25% holdout needs 4,995, which fits inside 5,200 with about 200 to spare, so she takes it.

Sincerely's launch check reaches the same verdict by slightly different arithmetic. For 5,200 accounts at a 25% holdout and a 6% baseline it reports that "this campaign can reliably detect a lift of 37% or larger. That is a well-powered design," and the Launch powered button is open. At the 10% holdout Sincerely recommends for lists of 1,000 accounts or more, the same list could only detect 52%, and a powered launch would be refused. The launch check uses a simpler approximation than the calculator, so near the line the two can disagree by a few hundred accounts. When they disagree, plan with the larger number.

## The arm sizes

Sincerely assigns arms by hashing the account ID, the campaign ID and a fixed salt into one of 10,000 buckets. An account whose bucket falls below 2,500 goes to the holdout. There is no random draw at launch. Anyone can recompute which side an account belongs on, and a retry cannot move it.

A hash split lands near the target, not exactly on it. Running Sincerely's assignment function over 5,200 made-up account IDs put 1,333 accounts in the holdout (25.6%) and 3,867 in the mailed arm. Those are the arm sizes for the rest of the example.

Held-out accounts are enrolled and walked through the same schedule as mailed ones. When a held-out account's letter comes due, the campaign log records `fulfillment suppressed: holdout` and no piece is created. The two arms differ in whether paper arrived and in nothing else.

## The counts when the window closes

Each account's 90 days start when it enrolls, or on the campaign's first mail date if that is later. An account enters the estimate only after its own window has closed, since a fleet enrolled last week has had no time to convert. Dana waits until every window has closed.

Delivery scans and landing-page replies both tempt teams to edit the data. The scans confirm 3,702 of the 3,867 letters, and the other 165 accounts stay in the mailed arm anyway. Dropping them would strip hard-to-reach fleets from one side only and flatter the result. Keeping them is called [intention to treat](https://trysincerely.com/glossary/intention-to-treat). A few fleets also reply through the letter's landing page. Only a mailed account can do that, so the report lists those replies beside the estimate as context and leaves them out of the lift. The opportunity counts come from the CRM, which records both arms the same way.

| Arm      | Accounts | Opportunities | Rate |
| -------- | -------- | ------------- | ---- |
| Mailed   | 3,867    | 317           | 8.2% |
| Held out | 1,333    | 79            | 5.9% |

A matchback report would stop here and credit the letter with 317 opportunities. The holdout says most of them were coming anyway. At the held-out rate of 5.9%, about 229 of the 3,867 mailed fleets would have opened an opportunity with no letter at all.

## The lift and its interval

Typed into the lift calculator, the four counts give:

| Figure                            | Value                         |
| --------------------------------- | ----------------------------- |
| Absolute lift                     | 2.27 percentage points        |
| 95% interval on the absolute lift | 0.74 to 3.81 points           |
| Relative lift                     | 38.3%                         |
| Incremental opportunities         | About 88, plausibly 28 to 147 |
| Two-sided p-value                 | 0.007                         |

Read the rows in that order. The absolute gain is what turns into pipeline: 2.27 points across 3,867 mailed accounts is about 88 extra opportunities, and the interval puts the true number anywhere from 28 to 147. The relative lift of 38% describes the same result against a small base. It sounds bigger, so it goes second and never alone.

Sincerely's report shows the same counts its own way and prints no p-value. Its headline is +38%, with a plausible range of +9% to +76%. That range is built as a risk ratio on the log scale, so it can never fall below -100%. Cost per incremental opportunity uses a separate Newcombe interval on the absolute difference, 0.65 to 3.73 points, which differs from the calculator's 0.74 to 3.81 only because the methods differ. The report is marked Ready to use, its mode chip reads powered, and its plain reading says: "Mailed accounts outperformed the holdout on opportunity created. The whole plausible range sits above zero."

## What Quarry Lane may say, and what it may not

Dana may say:

- Assigning fleets to the letter raised the 90-day opportunity rate from 5.9% to 8.2%, a gain of 2.3 points with a 95% range of about 0.7 to 3.8.
- That is roughly 88 extra opportunities across 3,867 accounts, plausibly as few as 28 or as many as 147.
- For this audience, this letter and this window, the direction is established, because the whole range sits above zero.

She may not say:

- "Direct mail lifts pipeline 38%." Without the counts and the range, the relative figure claims more precision than the test has.
- "The letter created 317 opportunities." That is matchback, and about 229 of those were coming anyway.
- Which opportunities the letter caused. A holdout estimates how many, never which ones.
- Anything about meetings, revenue, or what happens after day 90. The report's own caveat says it "says nothing about any outcome other than opportunity created, or about what happens after the 90-day window."
- That a particular design or line of copy did the work. The test compared one letter against no letter.

Base the budget decision on the low end of the range, not the midpoint. If 28 extra opportunities would not cover the letters, the lift is positive but the program has not yet shown that it pays. Sincerely reports cost per incremental opportunity as a range in credits for the same reason, and when the effect's range reaches zero it prints only a floor.

## A variant where the result is inconclusive

Dana repeats the test on 1,400 Canadian fleets with the same letter, preset, outcome and window. This time the launch check reads differently. At a 25% holdout it reports that "this campaign can reliably detect a lift of 74% or larger," then adds, "To detect a 40% lift you would need ~4,466 accounts." Raising the holdout to the 50% cap only brings that to 68%. Launch powered is unavailable, so she chooses Launch with pooling plan and keeps the 25% holdout.

The hash puts 366 accounts in the holdout and 1,034 in the mailed arm. Ninety days later:

| Arm      | Accounts | Opportunities | Rate |
| -------- | -------- | ------------- | ---- |
| Mailed   | 1,034    | 83            | 8.0% |
| Held out | 366      | 21            | 5.7% |

The lift calculator gives 2.29 points of absolute lift with a 95% interval of -0.61 to 5.19, a relative lift of 39.9%, and p = 0.15. Sincerely's report shows +40% with a plausible range of -12% to +122%, marks the report Still learning, and its mode chip reads "pooling: 1 of ~4 cohorts". Its plain reading: "Mailed accounts likely outperformed the holdout on opportunity created, but the range is wide and includes zero. Treat this as directional, not settled."

The midpoint is almost the same as the US midpoint. The conclusion is the opposite, because the interval is so much wider. With 366 held-out accounts, the test cannot tell a letter that does nothing from one that more than doubles the rate.

Dana may say that mailed fleets converted at 8.0% against 5.7%, that the direction is positive, and that this test cannot yet separate a 40% lift from none. She may not say that mail works in Canada, that it does not, or that it lifts opportunities 40%. "Not proven yet" and "no effect" are different sentences, and only the first one fits this result.

### What to do next

1. Leave the contract alone. Switching the outcome to meetings, or stretching the window to 120 days after seeing the numbers, turns a test into a search. Sincerely refuses both changes after launch.
2. Keep the holdout. Mailing all 1,400 fleets next quarter buys a bigger send and no evidence.
3. Run more cohorts with the same preset, outcome and window, each launched with a pooling plan. Sincerely pools campaigns that share all three and launched in pooling mode, then estimates one lift across them with a Mantel-Haenszel risk ratio. The US campaign does not join this pool, because it launched powered. At 1,400 accounts a cohort, the tracker expects about four.
4. Until then, report the counts and the range as descriptive evidence and label them that way. The guide to [measuring direct mail with a small audience](https://trysincerely.com/guides/direct-mail-small-audiences) covers what to put in that report.

### Four cohorts later

Suppose the next three Canadian cohorts come in at 79 of 1,045 mailed against 22 of 355 held out, then 88 of 1,073 against 23 of 327, then 76 of 1,052 against 19 of 348. Pooled with the first:

| Figure               | Value                                       |
| -------------------- | ------------------------------------------- |
| Mailed               | 326 opportunities from 4,204 accounts, 7.8% |
| Held out             | 85 opportunities from 1,396 accounts, 6.1%  |
| Pooled relative lift | +27%, plausible range +1% to +60%           |
| Absolute lift        | 1.67 points, range 0.09 to 3.08             |

The report moves to powered and Ready to use. The first cohort's 40% was a high draw. The pooled midpoint settled at 27%. The range now clears zero by a single point. That settles the direction but says little about the size. The low end of that range is about 4 extra opportunities across 4,204 accounts. Whether the Canadian program earns its postage depends on that low end, so the report should show it beside the midpoint.

## Check the numbers yourself

To reproduce the US power check, open the [holdout size calculator](https://trysincerely.com/tools/holdout-size) and enter a 6% baseline, a 40% lift, 80% power and a 25% holdout. To reproduce either readout, type the four counts into the [lift calculator](https://trysincerely.com/tools/lift-significance), or call the public endpoint, which reads no workspace data:

`GET /api/tools/lift?mailed=3867&mailedConversions=317&holdout=1333&holdoutConversions=79`

The [holdout testing guide](https://trysincerely.com/holdout-testing) explains each design decision in depth, and [how to prove outbound created pipeline](https://trysincerely.com/guides/how-to-prove-outbound-created-pipeline) carries a readout into a finance model. Sincerely's [measurement docs](https://trysincerely.com/docs/measurement) define the powered, pooling and descriptive labels used above.

## Related questions

- [Direct mail holdout and sample size calculator](https://trysincerely.com/tools/holdout-size): Size your direct mail holdout group. Enter baseline conversion rate and minimum detectable lift to get sample size at 80% or 90% power.
- [Direct mail incrementality and lift calculator](https://trysincerely.com/tools/lift-significance): Compare mailed and holdout conversion rates to get absolute and relative lift, a 95% confidence interval, and how many conversions were incremental.
- [How to prove outbound actually created pipeline](https://trysincerely.com/guides/how-to-prove-outbound-created-pipeline): Use an account-level randomized holdout, a fixed pipeline outcome, and intention-to-treat analysis to separate outbound attribution from causal lift.
- [How to measure direct mail with a small audience](https://trysincerely.com/guides/direct-mail-small-audiences): Keep an account-level holdout, predeclare one outcome, and pool comparable cohorts when your B2B direct mail audience is too small to measure alone.
- [What is a confidence interval?](https://trysincerely.com/glossary/confidence-interval): A confidence interval describes uncertainty around an estimate. Its width, direction, assumptions and business value all matter when acting on a result.

---

Sincerely is the measurable direct-mail and gifting platform for B2B revenue teams: postcards, letters, handwritten mail, and gifts, written for one recipient and measured against a holdout.

Contact Sincerely: https://trysincerely.com/contact

Agent routing index: https://trysincerely.com/llms.txt