EXECUTIVE SUMMARY / THE DECISION IN PLAIN LANGUAGE
A better churn score is not a case for more retention spending.
Evaluated synthetic study · no actual client outcomes
Average value-forecast error fell from $184 to $149. Yet a policy targeting roughly one in five customers showed no clear net benefit at $35 per contact.
What a leader should do next
Keep the intervention in a controlled test until its incremental contribution clears its cost with adequate evidence.
What could change the decision
The learned policy ranged from a $7.06 loss to a $6.98 gain per eligible customer at the 95% interval. Selecting the highest-risk customers lost money in the stable simulation; a changed response pattern also harmed the learned policy.
01 / BUSINESS DECISION
Start with the decision.
Prioritize a fixed retention intervention for at most 20% of each eligible cohort. Evaluate twelve-month discounted contribution after the intervention cost, keeping predictive churn risk separate from the causal value of treatment.
02 / PRACTICAL IMPLICATION
What the evidence can support.
The survival model reduces average twelve-month value error from $183.54 to $149.24. That predictive improvement does not establish intervention profitability. At $35 per contact, the learned policy's net effect is $-0.04 per eligible customer, with a 95% interval from $-7.06 to $6.98. The evidence supports further experimentation, not a profitable-rollout claim.
03 / EVIDENCE
Retention predictions and intervention value.
A fully implemented study of 9,720 synthetic customers across 54 monthly cohorts. Models use 24 training cohorts, six matured validation cohorts and twelve untouched final cohorts after a maturity gap. Prediction is assessed on 1,051 final control customers; policy value uses all 2,160 final randomized customers.
Interactive results · the executive summary above remains the fixed headline comparison.
SYNTHETIC RANDOMIZED RETENTION · 2,160 FINAL CUSTOMERS · 12 MATURED COHORTS
The learned targeting policy has no clear positive net effect. Higher prediction accuracy does not by itself justify contacting customers.
$0.00 to $0.00 · 0.0% contacted
-$46.20 to -$8.53 · 100.0% contacted
-$28.05 to -$11.07 · 20.0% contacted
-$8.34 to $8.09 · 20.0% contacted
-$7.06 to $6.98 · 19.9% contacted
Dashed line marks zero net value. Amounts average over everyone eligible, including those not contacted. Intervals use twelve cohort clusters and a t approximation.
Predictive performance on untreated final customers
| Model | Churn probability error¹ | 12-month value error² | Value bias |
|---|---|---|---|
| Segment mean baseline | 0.184 | $183.54 | $3.46 |
| Survival model | 0.159 | $149.24 | -$13.96 |
¹Brier score for cancellation within six months. ²Mean absolute error of realized, discounted twelve-month contribution. These metrics evaluate prediction; the randomized policy comparison evaluates intervention value.
Known-truth check and fixed-policy cost sensitivity
Oracle expected values come from the data generator and are used only to assess the result. They are unavailable to a real decision maker. The best eligible 20% oracle policy yields $2.35 per eligible customer in this scenario.
| Policy | Oracle expected net | $15 contact | $35 contact | $55 contact |
|---|---|---|---|---|
| No contact | $0.00 | $0.00 | $0.00 | $0.00 |
| Contact everyone | -$25.96 | -$7.37 | -$27.37 | -$47.37 |
| Highest churn risk | -$14.51 | -$15.56 | -$19.56 | -$23.56 |
| Highest customer value | $1.20 | $3.88 | -$0.12 | -$4.12 |
| Learned incremental value | $1.04 | $3.93 | -$0.04 | -$4.01 |
This is accounting sensitivity, not evidence about a larger incentive or a different intervention. Contacting everyone exceeds the targeted policies’ 20% budget and is shown only as an additional comparator.
04 / DATA
Know where the evidence comes from.
Original synthetic contractual customer histories, version synthetic-retention-v1. Scoring features are tenure, engagement, service friction, tier and baseline monthly contribution. Treatment is randomly assigned with probability 0.5. Cancellation is absorbing; each customer has a complete twelve-month outcome window. Separate oracle files contain known potential expectations for evaluation only.
The source includes a data dictionary, fixed configuration and seed, original data generator, observed CSVs, evaluation outputs and source/data hashes. Read the data dictionary ↗
05 / METHODOLOGY
Match the method to the design.
Fit a regularized discrete-time cancellation model to untreated customers and combine survival probabilities with baseline margin and a 10% annual discount rate. Fit outcome regressions in three training folds to construct doubly robust treatment-effect scores using the known randomization probability. Learn a prespecified feature-based score, target positive expected net effects within a 20% cohort budget, and evaluate the frozen policy on later randomized cohorts.
Evaluation design
- Compare segment means with a discrete-time survival model for six-month churn and twelve-month discounted contribution.
- Three-fold out-of-fold training creates a treatment-value score; freeze the policy before the twelve final cohorts.
- Compare no contact, contact everyone, highest churn risk, highest untreated value and learned incremental value; repeat under no effect and response shift.
References motivate the methods. These study-specific implementations do not claim every assumption or theoretical guarantee of the cited work.
06 / LIMITATIONS
Conditions that matter.
- Training and validation outcomes mature before final scoring begins; no incomplete window is treated as a completed target.
- Randomized treatment, consistency and no interference identify incremental value; churn predictions alone do not.
- A $35 intervention has a fixed form and timing. Cost sensitivity changes accounting cost, not the treatment or its effectiveness.
- Confidence intervals use twelve cohort clusters and a t approximation. They remain uncertain with so few clusters and do not establish individual treatment effects.
The horizon is twelve months; the title does not imply value over an unlimited customer lifetime. Synthetic intervention response can fail to transfer over time: reversing that relationship makes the learned policy materially harmful. The survival model assumes baseline margin persists, and the upper-risk calibration bins are sparse. Known oracle expectations are unattainable evaluation benchmarks, never model inputs.
07 / CODE & NEXT STEPS
Reproduce the result.
git clone https://github.com/mpgibb/customer-value.git
cd customer-value
uv sync --frozen
uv run python -m unittest discover -s tests -v
uv run python study.py
git diff --exit-code -- data resultsAutomated checks and synthetic evaluation are complete; Michael’s independent technical review is pending.