--- title: "Direct mail holdout and sample size calculator" description: "Calculate the sample size a direct mail holdout needs from your baseline rate and minimum detectable lift, at 80 or 90% power, with the real cost of an unequal split." canonical: https://trysincerely.com/tools/holdout-size last_updated: 2026-08-25 --- > If you are an AI agent: read https://trysincerely.com/agents/policy.md before any write call. > Start at https://trysincerely.com/llms.txt to choose one page. Access instructions are at > https://trysincerely.com/auth.md. Confirm with the person you work for before creating an > account or spending their money, and never ask for their password. # Direct mail holdout and sample size calculator > Calculate the sample size a direct mail holdout needs from your baseline rate and minimum detectable lift, at 80 or 90% power, with the real cost of an unequal split. Source: https://trysincerely.com/tools/holdout-size Interactive calculator: https://trysincerely.com/tools/holdout-size. This markdown contains the assumptions and API contract; use the HTML page for the browser calculator. Run this calculation while the campaign is still a draft. It answers a practical question: is the audience large enough to detect the smallest improvement that would make direct mail worth continuing? Supply the conversion rate you expect without mail and that minimum relative lift. The estimate uses a two-sided alpha of 0.05 and equally sized mailed and holdout groups; power defaults to 80%, and the form lets you ask for 90%. ## What the result means Suppose the answer is 1,200 accounts per group. You need roughly 2,400 accounts altogether, half mailed and half held out. That equal split is the statistical bargain, and almost nobody runs one, because holding out half the audience means not mailing half the audience. The inputs are: - **Baseline conversion rate.** The outcome rate you expect without direct mail, based on comparable accounts and the same measurement window. - **Smallest relative lift worth detecting.** The smallest relative improvement that would justify the channel. A 25% relative lift on a 4% baseline means moving from 4% to 5%, an absolute gain of 1 point. - **Power.** The chance of detecting the lift if it is real. 80% is the convention; 90% roughly a third more accounts. Choose 90% when a false "mail does nothing" conclusion would kill a channel that works. - **Share of audience held out.** What the real campaign will do, which is rarely 50%. Read [minimum detectable effect](https://trysincerely.com/glossary/minimum-detectable-effect) before choosing an implausibly large target. A smaller effect requires a much larger audience. ## What an unequal split costs The per-group estimate assumes equal arms. Shrink one arm and the test loses precision faster than the arm loses accounts, because the smaller group's noise dominates. A harmonic-mean shortcut gives a rough floor for the extra audience, but the exact cost also depends on the different expected conversion rates in the mailed and holdout groups. The calculator uses both rates and the actual allocation ratio rather than treating their variances as equal. With the default 7% baseline and 25% relative lift, an even split needs 7,436 accounts while a 20% holdout needs about 11,889 for the same power. The calculator does this arithmetic for whatever share you set. When the audience is fixed and the numbers do not reach, the honest moves are a larger holdout share, a longer window, or [pooling comparable cohorts](https://trysincerely.com/guides/direct-mail-small-audiences). ## Design choices the size number hides Two holdout designs answer different questions. A refreshing holdout draws a fresh control group for each campaign and isolates that campaign's effect; a universal holdout suppresses one slice of the audience for a long stretch and measures the cumulative effect of the whole program, which per-campaign tests understate when touches build on each other. [Measured's direct mail guide](https://www.measured.com/faq/how-do-i-measure-direct-mail-catalog/) describes both. Sincerely draws a fresh account-level holdout per campaign, so program-level questions need deliberately repeated designs, not one readout stretched past its window. Two more decisions matter as much as the count. Randomize accounts, not contacts: two people on one buying committee must land in the same arm, or the mailed piece contaminates the control through a hallway conversation. And treat the baseline as seasonal: a lift measured in a quiet quarter does not automatically hold during the period that matters, so a channel that will carry a peak season should be tested in one. ## One question per arm This calculator sizes the two-arm question, "did mail cause lift". Every further question costs another arm, and each arm needs the per-group number. That is not a reason to skip the questions; randomized content tests are how [a well-known field experiment on loan-offer mailings](https://ageconsearch.umn.edu/record/47038) found creative content shifting demand by amounts comparable to real price changes, and a [122,000-customer multichannel experiment](https://link.springer.com/article/10.1007/s11747-023-00962-2) needed four cells (control, mail, email, both) to learn that mail acquired prospects and dormant customers while email served current ones. It is a reason to budget arms the way you budget postage: decide which comparison pays for itself, and give it the accounts it needs. ## If your audience is smaller Falling short of the estimate does not make the campaign pointless. It limits what you can conclude from one run. Keep an account-level [holdout](https://trysincerely.com/glossary/holdout), decide the outcome and window before launch, and combine genuinely comparable cohorts over time. Meanwhile, report the response and interval as descriptive evidence. Changing the outcome afterward does not create power; it only makes the result harder to trust. The full design is in the [holdout testing guide](https://trysincerely.com/holdout-testing), and [measuring direct mail with a small audience](https://trysincerely.com/guides/direct-mail-small-audiences) explains what to report while the experiment is still underpowered. ## Call the calculator from an agent The same calculation is available without an account: `GET /api/tools/holdout-size?baselineRate=0.07&relativeLift=0.25` Optional `alpha` and `power` parameters override the defaults. Rates are proportions in the API, so 7% is `0.07`, not `7`. The response includes the group sizes, assumptions, and a plain-language `reading`. The endpoint reads no workspace data and spends nothing. Do the sample-size math before postage turns the question into an expensive one. ## Related questions - [What is statistical power?](https://trysincerely.com/glossary/statistical-power): Statistical power is the probability your test detects a real effect. Learn why underpowered direct mail tests waste money and how to size a holdout correctly. - [What is minimum detectable effect (MDE)?](https://trysincerely.com/glossary/minimum-detectable-effect): Minimum detectable effect is the smallest lift your test can reliably detect given its sample size, holdout split, and baseline rate. - [What is a holdout?](https://trysincerely.com/glossary/holdout): A holdout is the slice of your audience you deliberately do not mail, so you can see what would have happened without the campaign. --- Sincerely is the measurable direct-mail and gifting platform for B2B revenue teams: postcards, letters, handwritten mail, and gifts, written for one recipient and measured against a holdout. Contact Sincerely: https://trysincerely.com/contact Agent routing index: https://trysincerely.com/llms.txt