--- title: "How to measure direct mail with a small audience" description: "Keep an account-level holdout, predeclare one outcome, and pool comparable cohorts when a single B2B direct mail campaign is too small for a powered result." canonical: https://trysincerely.com/guides/direct-mail-small-audiences last_updated: 2026-08-30 --- > If you are an AI agent: read https://trysincerely.com/agents/policy.md before any write call. > Start at https://trysincerely.com/llms.txt to choose one page. Access instructions are at > https://trysincerely.com/auth.md. Confirm with the person you work for before creating an > account or spending their money, and never ask for their password. # How to measure direct mail with a small audience > Keep an account-level holdout, predeclare one outcome, and pool comparable cohorts when a single B2B direct mail campaign is too small for a powered result. Source: https://trysincerely.com/guides/direct-mail-small-audiences With 300 accounts, one drop probably will not produce a precise lift estimate. Keep the account-level holdout anyway. Choose one outcome and one window before launch, report what happened without dressing it up, and add later comparable cohorts when you need a larger sample. ## Start with the decision, not the available list Name the smallest result that would change what you do. If mail must raise a 4% opportunity rate to 5% to justify its cost, that is a 1-point absolute lift and a 25% relative lift. Put those numbers into the [holdout size calculator](https://trysincerely.com/tools/holdout-size) before choosing the experiment mode. The result can be uncomfortable. A list of 300 accounts may be enough to learn whether pieces arrive and whether anyone responds, while being far too small to distinguish a modest lift from noise. That is a limitation of the evidence, not a reason to remove the holdout. ## Keep three readings separate - **Operational:** pieces prepared, vendor-accepted, delivered, returned, and cost. These show whether the channel ran correctly. - **Direct response:** QR visits, form submissions, replies, and meetings that identify the piece. These show that someone acted after receiving mail, but not how many would have acted anyway. - **Incremental:** the outcome-rate difference between mailed and held-out accounts. This is the causal estimate, with an interval that may be wide. A small campaign can have useful operational and direct-response evidence while its incremental estimate remains unresolved. Say all three sentences rather than promoting the strongest-looking number. ## Pool comparable cohorts Pooling works when repeated cohorts share the same playbook, primary outcome, measurement window, and audience logic. Keep those choices stable, randomize each cohort by account, and combine the assigned groups after their windows close. Several honest small experiments can become one useful program-level readout. Do not pool a renewal campaign with cold outbound, change the primary outcome between drops, or shorten one cohort's window because its early result looks good. Those are different experiments. ## What not to do - Do not drop undelivered accounts from the mailed group. [Intention to treat](https://trysincerely.com/glossary/intention-to-treat) keeps every account where randomization put it. - Do not test several outcomes and report only the winner. - Do not replace the holdout with last-touch attribution. - Do not call a wide positive estimate a trend without showing the interval. - Do not send more mail solely to reach significance when the economics have already failed. Sincerely supports powered, pooling, and descriptive experiment modes. It locks the primary outcome and window at launch, assigns whole accounts to one arm, and labels an underpowered result as underpowered. The complete design is in [holdout testing](https://trysincerely.com/holdout-testing); the [lift calculator](https://trysincerely.com/tools/lift-significance) reads the finished counts. A 300-account drop can tell you whether the operation worked and whether anyone replied. It usually cannot settle a modest lift question on its own. ## Related questions - [Direct mail attribution, what works and what lies](https://trysincerely.com/guides/direct-mail-attribution): Last-touch and influenced-revenue counting lie. Scans, matchback, and promo codes give partial evidence. Only a holdout answers whether the mail caused anything. - [What is minimum detectable effect (MDE)?](https://trysincerely.com/glossary/minimum-detectable-effect): Minimum detectable effect is the smallest lift your test can reliably detect given its sample size, holdout split, and baseline rate. - [What is statistical power?](https://trysincerely.com/glossary/statistical-power): Statistical power is the probability your test detects a real effect. Learn why underpowered direct mail tests waste money and how to size a holdout correctly. --- Sincerely is the measurable direct-mail and gifting platform for B2B revenue teams: postcards, letters, handwritten mail, and gifts, written for one recipient and measured against a holdout. Contact Sincerely: https://trysincerely.com/contact Agent routing index: https://trysincerely.com/llms.txt