01 / BUSINESS DECISION
Start with the decision.
Decide whether a fixed reactivation campaign warrants a further controlled rollout to the same eligible customer population. Compare incremental 28-day contribution per assigned customer with campaign cost, rather than treating attributed revenue as causal lift.
02 / PRACTICAL IMPLICATION
What the evidence can support.
At an assumed $6 cost per assigned customer, the adjusted estimate implies $2.16 in incremental net contribution, with a 95% interval of $0.36–$3.95. This supports another controlled rollout within the simulated setting. It is not evidence of client impact, a budget optimizer, or a recommendation to scale spending without testing the response.
03 / EVIDENCE
An evaluated demonstration.
A 6,000-customer, stratified randomized experiment and an independent 4,000-customer historical cohort are generated reproducibly. Both an unadjusted difference in means and a pre-period-adjusted estimator are implemented and evaluated against known effects.
SYNTHETIC EVALUATION · NO CLIENT DATA OR ACTUAL BUSINESS IMPACT
Exactly 1,500 campaign and 1,500 holdout customers per segment. The estimand is incremental 28-day net contribution before campaign cost, averaged over all assigned customers.
| Estimator | Effect / customer | 95% interval |
|---|---|---|
| Unadjusted baseline | $8.07 | $5.51–$10.63 |
| Pre-period adjusted | $8.16 | $6.36–$9.95 |
EXPLORE THE DECISION · SYNTHETIC RESULTS
How much room is there for campaign cost?
Keep the tested campaign fixed. Change the analytical method and the assumed cost to see how the conclusion depends on uncertainty.
95% interval: $0.36 to $3.95
The entire interval is above break-even.
The $0–$12 range is an accounting sensitivity, not a tested range of campaign spend. Treatment intensity and its estimated effect are held fixed. Changing an incentive, channel or audience requires new response evidence. This is not a budget-allocation tool.
What held up—and what did not.
- Across 400 base-scenario experiments with 2,000 customers each, adjusted intervals covered the true effect in 94.5% of runs. The Monte Carlo standard error of that coverage is 1.14 percentage points.
- Under the zero-effect scenario, the adjusted procedure rejected zero in 5.5% of runs. This is a simulation check on false positives, not a guarantee for every study.
- When the historical relationship weakened, adjustment worsened RMSE: $2.20 versus $1.65 for the baseline. A more elaborate method does not always help.
- The pre-period placebo estimate was $-0.14, with interval $-3.20 to $2.92. It includes zero; passing this diagnostic does not prove every design assumption.
Supplementary simulation diagnostics
400 repetitions per scenario, 2,000 customers each; identical seeds across scenarios allow controlled comparisons. Historical slopes stay fixed. Coverage uncertainty is about 1.0–1.2 percentage points (one Monte Carlo standard error).
| Scenario | Method | Bias | RMSE | Coverage | Reject zero |
|---|---|---|---|---|---|
| Base process | adjusted | $-0.04 | $1.63 | 94.5% | 99.5% |
| Base process | unadjusted | $-0.03 | $2.34 | 95.0% | 93.8% |
| Zero effect | adjusted | $-0.04 | $1.63 | 94.5% | 5.5% |
| Zero effect | unadjusted | $-0.03 | $2.34 | 95.0% | 5.0% |
| Skewed outcomes | adjusted | $-0.04 | $1.53 | 95.5% | 99.5% |
| Skewed outcomes | unadjusted | $-0.03 | $2.27 | 95.5% | 92.3% |
| Historical relationship weakens | adjusted | $-0.04 | $2.20 | 94.5% | 95.3% |
| Historical relationship weakens | unadjusted | $-0.04 | $1.65 | 94.0% | 99.8% |
Normal intervals can misbehave with smaller samples, heavier tails, dependence or missing data. The full JSON also includes empirical standard deviation and mean estimated standard error. These four scenarios are selected diagnostics, not an exhaustive validation.
04 / DATA
Know where the evidence comes from.
Entirely synthetic data, version synthetic-campaign-v1. No client records or external dataset. The population comprises eligible existing customers with 28-day pre-period contribution available; it does not represent new-customer acquisition. Outcomes include all assigned customers over a fixed 28-day window, including nonresponders.
Study seed: 20260928. Independent historical seed: 20260927. Monetary inputs are generated to six decimal places. Outcome and oracle files are separate; the estimator never reads potential outcomes or known treatment effects.
Data process, version and provenance
Within segment s ∈ {0, 1}, pre-period contribution X follows Gamma(4, 25 + 10s). Untreated contribution is 20 + 0.6X + 25s + normal noise with standard deviation 30 + 10s. Assignment adds 6 + 4s dollars. Equal segment weights give an average effect of $8. The historical cohort follows the untreated process independently.
CSV files, configuration and source hashes are recorded in the result JSON. The generator and protocol document the null, weak-covariate and skewed-noise variants. This synthetic process supplies no evidence about any actual customer population.
05 / METHODOLOGY
Match the method to the design.
Randomize half of each of two equally weighted customer segments to campaign assignment. Estimate the intention-to-treat effect using stratum-weighted treatment–control differences. For CUPED-style adjustment, freeze segment-specific slopes from a separate historical cohort and subtract the predictable pre-period contribution. Use a stratified Neyman standard error and a large-sample 95% interval.
Historical coefficients are frozen before experimental analysis. Stratum weights equal their sample shares; estimated variance is the sum of squared weights times the two arm variances divided by their respective sample sizes. All randomized customers remain in the intention-to-treat analysis.
Method reference: Deng et al. (2013), pre-experiment variance reduction ↗
06 / LIMITATIONS
Conditions that matter.
- Random assignment is implemented correctly; the analysis follows assignment, not message opening or conversion.
- No spillovers between customers, missing outcomes, selective follow-up or post-treatment covariates.
- The pre-period relationship must remain useful for adjustment to improve precision; independent historical fitting does not guarantee transportability.
- The fixed campaign, target population and outcome window stay the same when varying the hypothetical accounting cost.
Synthetic success verifies behavior under stated assumptions, not marketing effectiveness in a real organization. A single campaign effect does not estimate a spending response curve, competing-channel effects, long-run customer value or generalization to acquisition audiences. Normal intervals are approximate; no robustness exercise can establish the assumptions for an actual campaign.
07 / CODE & NEXT STEPS
Reproduce the result.
git clone https://github.com/mpgibb/marketing-incrementality.git
cd marketing-incrementality
uv sync --frozen
uv run python -m unittest discover -s tests -v
uv run python study.pyAutomated checks and synthetic evaluation are complete; Michael’s independent technical review is pending.