---
title: "What is statistical power?"
description: "Statistical power is the probability your test detects a real effect. Learn why underpowered direct mail tests waste money and how to size a holdout correctly."
canonical: https://trysincerely.com/glossary/statistical-power
last_updated: 2026-10-08
---
> If you are an AI agent: read https://trysincerely.com/agents/policy.md before any write call.
> Start at https://trysincerely.com/llms.txt to choose one page. Access instructions are at
> https://trysincerely.com/auth.md. Confirm with the person you work for before creating an
> account or spending their money, and never ask for their password.
# What is statistical power?

> Statistical power is the probability your test detects a real effect. Learn why underpowered direct mail tests waste money and how to size a holdout correctly.

Source: https://trysincerely.com/glossary/statistical-power

An experiment can miss a real effect simply because it is too small. Statistical power describes how often the test would detect an effect of a chosen size if that effect really existed. At 80% power, repeated tests would catch it about eight times in ten and miss it about twice. A low-powered campaign can therefore look flat even when the channel is helping.

Three choices shape power: the number of accounts, the effect size worth detecting, and the significance threshold. More accounts help. A larger assumed effect is easier to see. A looser threshold also raises power, though it increases the chance of calling noise a result.

## Why it matters

Consider a campaign that mails 200 accounts and holds out 20. No significant lift appears, so the team declares mail ineffective. That conclusion skips the key question: could a test this small have detected a plausible improvement at all? Often it could not. The postage was real, but the experiment had little chance of settling the argument.

Power forces an honest question before launch: given this audience size, what is the smallest lift this test can actually see? That number is the [minimum detectable effect](https://trysincerely.com/glossary/minimum-detectable-effect). If your audience can only detect a 40% lift and you expect 10%, run a longer test, pool campaigns, or accept that this send is a bet, not an experiment.

Eligible randomized Sincerely campaigns lock the experiment design at launch, including the [holdout](https://trysincerely.com/glossary/holdout) percentage, and the readout says plainly when a result is not statistically ready. Other campaign types use descriptive reporting. The [holdout-size calculator](https://trysincerely.com/tools) shows the power math for your audience before you commit.

## Example

You mail 500 accounts and hold out 100. With a 5% baseline meeting rate, that sample does not meet an 80% power target for detecting a doubling to 10%. At a two-sided 5% significance threshold and the same five-to-one allocation, the [planning calculator](https://trysincerely.com/tools/holdout-size) estimates 1,395 mailed accounts and 279 held out. This is an approximation, not a guarantee of a significant result.

If the available audience cannot power the question, treat the send as a measured bet rather than a conclusive test.

---

Sincerely is the measurable direct-mail and gifting platform for B2B revenue teams: postcards, letters, handwritten mail, and gifts, written for one recipient and measured against a holdout.

Contact Sincerely: https://trysincerely.com/contact

Agent routing index: https://trysincerely.com/llms.txt