EXECUTIVE SUMMARY / THE DECISION IN PLAIN LANGUAGE
The campaign clears its assumed cost in this experiment.
Evaluated synthetic study · no actual client outcomes
The simulated campaign added an estimated $8.16 per assigned customer before campaign cost. At a $6 cost, that leaves $2.16 in incremental net contribution, with a 95% interval of $0.36–$3.95.
What a leader should do next
Use this design for another controlled test in the same eligible population, with a contribution threshold agreed before results are known.
What could change the decision
These are synthetic customers and a fixed campaign. The result does not show what extra spending, a different offer or a new audience would produce. Adjustment also becomes less precise when the historical relationship weakens.
01 / BUSINESS DECISION
Start with the decision.
Decide whether a fixed reactivation campaign warrants a further controlled rollout to the same eligible customer population. Compare incremental 28-day contribution per assigned customer with campaign cost, rather than treating attributed revenue as causal lift.
02 / PRACTICAL IMPLICATION
What the evidence can support.
At an assumed $6 cost per assigned customer, the adjusted estimate implies $2.16 in incremental net contribution, with a 95% interval of $0.36–$3.95. This supports another controlled rollout within the simulated setting. It is not evidence of client impact, a budget optimizer, or a recommendation to scale spending without testing the response.
03 / EVIDENCE
Campaign effect and cost uncertainty.
A 6,000-customer, stratified randomized experiment and an independent 4,000-customer historical cohort are generated reproducibly. Both an unadjusted difference in means and a pre-period-adjusted estimator are implemented and evaluated against known effects.
SYNTHETIC EVALUATION · NO CLIENT DATA OR ACTUAL BUSINESS IMPACT
Exactly 1,500 campaign and 1,500 holdout customers per segment. The estimand is incremental 28-day net contribution before campaign cost, averaged over all assigned customers.
| Estimator | Effect / customer | 95% interval |
|---|---|---|
| Unadjusted baseline | $8.07 | $5.51–$10.63 |
| Pre-period adjusted | $8.16 | $6.36–$9.95 |
Interactive results · the executive summary above remains the fixed headline comparison.
EXPLORE THE DECISION · SYNTHETIC RESULTS
How much room is there for campaign cost?
Keep the tested campaign fixed. Change the analytical method and the assumed cost to see how the conclusion depends on uncertainty.
95% interval: $0.36 to $3.95
The entire interval is above break-even.
The $0–$12 range is an accounting sensitivity, not a tested range of campaign spend. Treatment intensity and its estimated effect are held fixed. Changing an incentive, channel or audience requires new response evidence. This is not a budget-allocation tool.
What held up—and what did not.
- Across 400 base-scenario experiments with 2,000 customers each, adjusted intervals covered the true effect in 94.5% of runs. The Monte Carlo standard error of that coverage is 1.14 percentage points.
- Under the zero-effect scenario, the adjusted procedure rejected zero in 5.5% of runs. This is a simulation check on false positives, not a guarantee for every study.
- When the historical relationship weakened, adjustment worsened RMSE: $2.20 versus $1.65 for the baseline. A more elaborate method does not always help.
- The pre-period placebo estimate was -$0.14, with interval -$3.20 to $2.92. It includes zero; passing this diagnostic does not prove every design assumption.
Supplementary simulation diagnostics
400 repetitions per scenario, 2,000 customers each; identical seeds across scenarios allow controlled comparisons. Historical slopes stay fixed. Coverage uncertainty is about 1.0–1.2 percentage points (one Monte Carlo standard error).
| Scenario | Method | Bias | RMSE | Coverage | Reject zero |
|---|---|---|---|---|---|
| Base process | adjusted | -$0.04 | $1.63 | 94.5% | 99.5% |
| Base process | unadjusted | -$0.03 | $2.34 | 95.0% | 93.8% |
| Zero effect | adjusted | -$0.04 | $1.63 | 94.5% | 5.5% |
| Zero effect | unadjusted | -$0.03 | $2.34 | 95.0% | 5.0% |
| Skewed outcomes | adjusted | -$0.04 | $1.53 | 95.5% | 99.5% |
| Skewed outcomes | unadjusted | -$0.03 | $2.27 | 95.5% | 92.3% |
| Historical relationship weakens | adjusted | -$0.04 | $2.20 | 94.5% | 95.3% |
| Historical relationship weakens | unadjusted | -$0.04 | $1.65 | 94.0% | 99.8% |
Normal intervals can misbehave with smaller samples, heavier tails, dependence or missing data. The full JSON also includes empirical standard deviation and mean estimated standard error. These four scenarios are selected diagnostics, not an exhaustive validation.
04 / DATA
Know where the evidence comes from.
Entirely synthetic data, version synthetic-campaign-v1. No client records or external dataset. The population comprises eligible existing customers with 28-day pre-period contribution available; it does not represent new-customer acquisition. Outcomes include all assigned customers over a fixed 28-day window, including nonresponders.
Study seed: 20260928. Independent historical seed: 20260927. Monetary inputs are generated to six decimal places. Outcome and oracle files are separate; the estimator never reads potential outcomes or known treatment effects.
Data process, version and provenance
Within segment s ∈ {0, 1}, pre-period contribution X follows Gamma(4, 25 + 10s). Untreated contribution is 20 + 0.6X + 25s + normal noise with standard deviation 30 + 10s. Assignment adds 6 + 4s dollars. Equal segment weights give an average effect of $8. The historical cohort follows the untreated process independently.
CSV files, configuration and source hashes are recorded in the result JSON. The protocol documents null, weak-covariate and skewed-noise variants. This process supplies no evidence about an actual customer population.
05 / METHODOLOGY
Match the method to the design.
Randomize half of each of two equally weighted customer segments to campaign assignment. Estimate the intention-to-treat effect using stratum-weighted treatment–control differences. For CUPED-style adjustment, freeze segment-specific slopes from a separate historical cohort and subtract the predictable pre-period contribution. Use a stratified Neyman standard error and a large-sample 95% interval.
Historical coefficients are frozen before experimental analysis. Stratum weights equal their sample shares; estimated variance sums squared weights times the two arm variances divided by their sample sizes. All randomized customers remain in the analysis.
06 / LIMITATIONS
Conditions that matter.
- Random assignment is implemented correctly; the analysis follows assignment, not message opening or conversion.
- No spillovers between customers, missing outcomes, selective follow-up or post-treatment covariates.
- The pre-period relationship must remain useful for adjustment to improve precision; independent historical fitting does not guarantee transportability.
- The fixed campaign, target population and outcome window stay the same when varying the hypothetical accounting cost.
Synthetic success verifies behavior under stated assumptions, not marketing effectiveness in a real organization. A single campaign effect does not estimate a spending response curve, competing-channel effects, long-run customer value or generalization to acquisition audiences. Normal intervals are approximate; no robustness exercise can establish the assumptions for an actual campaign.
07 / CODE & NEXT STEPS
Reproduce the result.
git clone https://github.com/mpgibb/marketing-incrementality.git
cd marketing-incrementality
uv sync --frozen
uv run python -m unittest discover -s tests -v
uv run python study.py
git diff --exit-code -- data resultsAutomated checks and synthetic evaluation are complete; Michael’s independent technical review is pending.