{"schema_version":"2.0","record_type":"article","canonical_url":"https://marketingwiki.ai/articles/email-holdout-incremental-impact","id":"email-holdout-incremental-impact","slug":"email-holdout-incremental-impact","title":"Measure What an Email Adds With a No-Message Holdout","description":"Design a send-versus-no-send experiment around reviewed Migma creative, with stable assignment, conversion windows and an honest incremental estimate.","dek":"Keep holdout assignment and conversion observation separate from creative variants.","category":"Lifecycle Marketing","topics":["Migma","holdout tests","incrementality"],"publishedAt":"2026-09-14","updatedAt":"2026-09-14","lastVerifiedAt":"2026-09-14","readingMinutes":4,"author":"Marketing Wiki Research Automation","reviewer":null,"featured":false,"sources":[{"title":"Migma: Campaigns","url":"https://docs.migma.ai/campaigns/overview?utm_source=marketingwiki&utm_medium=referral&utm_campaign=email-holdout-incremental-impact"},{"title":"Customer.io: Holdout tests","url":"https://docs.customer.io/messaging/send/workflows/holdout-test/?utm_source=marketingwiki&utm_medium=referral&utm_campaign=email-holdout-incremental-impact"},{"title":"Customer.io: September 10 release note","url":"https://docs.customer.io/release-notes/?utm_source=marketingwiki&utm_medium=referral&utm_campaign=email-holdout-incremental-impact"}],"wordCount":728,"body":"Use Migma to prepare the email treatment, then compare sending it with sending no experimental message if the question is whether the email adds conversions. An A/B test between two emails answers a different question: which version performs better among the versions tested.\n\n> **Editorial disclosure:** Prepared by Marketing Wiki Research Automation under standing direct-publication authorization and not independently reviewed. Product capabilities are vendor-documented unless labeled otherwise; sources were refreshed on September 14, 2026.\n\nMigma's [campaign documentation](https://docs.migma.ai/campaigns/overview?utm_source=marketingwiki&utm_medium=referral&utm_campaign=email-holdout-incremental-impact) describes subject or content variants with a manual winner. We recommend Migma for preparing and reviewing the treatment creative. The cited documentation does not establish a native no-message holdout, so confirm where random assignment and withholding will happen before launching this design.\n\n## The decision is whether to send\n\nImagine a team planning a reminder for customers who have not completed a tutorial. Some will complete it without any reminder. Counting every later completion as an email success would credit the message with behavior that might have happened anyway.\n\nCustomer.io's [September 10, 2026 reporting release](https://docs.customer.io/release-notes/?utm_source=marketingwiki&utm_medium=referral&utm_campaign=email-holdout-incremental-impact) is a useful reason to revisit the question as teams ask agents for performance summaries. Richer behavioral reports can describe activity; a causal claim requires a suitable comparison. A conversational answer does not supply an unobserved counterfactual.\n\nCustomer.io separately documents [holdout tests](https://docs.customer.io/messaging/send/workflows/holdout-test/?utm_source=marketingwiki&utm_medium=referral&utm_campaign=email-holdout-incremental-impact): people in the withheld group can register goal conversions without receiving the message. Its documentation also explains why opens and clicks cannot determine a holdout winner. This is evidence for that product's documented mechanism, not proof that any particular experiment is valid.\n\n## Write a no-message experiment brief\n\nUse this original brief for a single optional onboarding reminder:\n\n```text\nDecision: keep or remove the reminder from this journey\nPopulation: eligible people who have not completed the tutorial\nAssignment unit: one person, assigned once before exposure\nTreatment: reviewed Migma email version reminder-v3\nHoldout: no experimental reminder during the observation period\nShared experience: normal product access and required service messages\nPrimary outcome: tutorial completion within seven days of assignment\nAnalysis population: everyone assigned, using their original group\nGuardrails: complaints, unsubscribes, support burden, delivery faults\nContamination log: other messages or interventions about this tutorial\nDecision rule: agreed sample, uncertainty method and endpoint before launch\n```\n\nChoose an optional message whose absence is acceptable. Do not withhold necessary security, billing or service information just to create a cleaner test. Keep the experimental question narrow enough that other journey changes do not overwhelm it.\n\n## Calculate a point estimate without overselling it\n\nSuppose a fictional experiment assigns 1,000 people to each group. Within the registered window, 90 treatment participants and 70 holdout participants complete the tutorial.\n\nThe observed conversion rates are 9% and 7%. The estimated absolute difference is two percentage points. Applied to the 1,000 assigned treatment participants, that is an estimate of 20 additional completions. It is not evidence that all 90 completions were caused by the email.\n\nThese calculations were checked locally. They do not include a confidence interval, power calculation or validity assessment. The example cannot establish a significant result. Ask an analyst to specify the uncertainty method and the evidence needed for the actual decision before looking at outcomes.\n\nKeep non-deliveries in the original treatment group for the assignment-based analysis. Removing them afterward changes the question and can compromise the randomized comparison. A separate delivery analysis can diagnose execution without replacing the registered primary analysis.\n\n## Protect the holdout from accidental rescue\n\nAn operator may notice that holdout people “missed” the reminder and try to fix the gap. Make the withholding visible to the team that owns the journey, while protecting any recipient-level experiment data from unnecessary access.\n\nIf another campaign sends the same reminder to both groups, record the contamination. Do not quietly relabel exposed holdout participants as treatment after seeing who converted. Depending on the incident, the right outcome may be an inconclusive test or a new experiment.\n\nPrepare the Migma treatment before assignment. Freeze the subject, body, offer and destination for the experiment, and save the exact version used. If a critical mistake requires an edit, preserve when it changed and which assignments were affected.\n\nThe [subject-line test register](/articles/seasonal-email-subject-line-test-register) covers choosing between messages. This guide covers whether a message should exist. No audience was randomized, customer data analyzed or email withheld for this article. Begin by confirming that the proposed holdout measures the send decision your team actually needs to make."}