Chapter 04 of 10
What the data can identify
Collinearity after transformation, degrees of freedom, and baseline flexibility — the three ways an MMM returns your prior and calls it a finding.
Marketing mix modelling asks a causal question of observational data. Nothing was randomized, budgets were set by people responding to demand, and channels were bought together. So the honest first question is not “what is the ROI” but whether this design can identify one at all. When it cannot, a Bayesian model does not fail loudly — it returns the prior, with a credible interval, looking exactly like a finding.
Three mechanisms produce that failure, and the engine measures all three before fitting.
Collinearity, after transformation
Two channels that move together cannot be told apart, and the model will trade their coefficients off along a ridge while total media contribution stays perfectly well-identified. The measurement has to happen on hill(adstock(spend)) evaluated at the prior medians, because that — not raw spend — is what the likelihood sees. Adstock is a smoother, so two channels can be visibly different week to week and nearly identical after carryover.
Variance inflation is computed the same way, regressing each transformed channel on the others plus the nuisance block — spline basis, geo dummies and controls — so that a channel collinear with seasonality rather than with another channel is also caught.
source · engine/mmm/diagnose.py
Conditioning and degrees of freedom
The condition number of the media block answers a question no pairwise correlation can: whether some combination of channels is nearly degenerate even though no two of them are. Degrees of freedom is the crudest check and often the decisive one — a five-year weekly national panel is 260 observations, and a six-channel model with a 20-knot baseline and four controls is already spending a large fraction of them.
| Check | Measures | Trips at | Reading |
|---|---|---|---|
| History | Number of periods | < 104 · < 52 | Under 52 periods an annual seasonal pattern cannot be identified at all. Between 52 and 104 there is one pass through the cycle, and seasonality and media are hard to separate. |
| Degrees of freedom | (geos × periods) / parameters | < 4× · < 2× | Chan & Perry’s rule of thumb. Below 4× expect wide intervals; below 2× expect the prior back. |
| Correlation | Worst pair, post-transform | |r| > 0.8 · > 0.9 | Above 0.9 the two channels are near-duplicates and no model can attribute between them. |
| VIF | Against all others plus nuisance | > 5 · > 10 | Above 10 the channel is nearly a linear combination of the other regressors. |
| Condition number | Media block, nuisance projected out | > 30 · > 100 | Above 100 the design is close to singular. |
| Per-channel variation | Coefficient of variation of spend | < 0.15 · < 0.05 | A channel whose spend never moves has nothing to correlate with the KPI. Its ROI comes back as whatever the prior said. |
Baseline flexibility is the third mechanism
Knot density is the single most important identification dial in an MMM, and it is usually set by default and never revisited. A baseline flexible enough to track the KPI week by week will absorb the media signal and collapse every ROI onto its prior — and the model will fit beautifully while doing it, because in-sample fit is exactly what extra knots buy.
n_knots = max(4, round(n_periods / 20))
The choice here is deliberately conservative. Users can raise it, but they have to choose to, and the readiness report counts knots against degrees of freedom so the cost is visible at the moment it is paid.
Confounding the design cannot fix
Two structural problems survive every check above, because they are about which columns exist rather than how they covary.
No demand proxy.Media budgets follow demand. Without a control that carries the demand signal — category volume, distribution, a search-interest index — every channel’s ROI absorbs the correlation between spending more and selling more anyway, and reads high.
Post-treatment mediators. A column downstream of media on the causal path — site sessions, branded search volume, app installs — absorbs the very effect being estimated when conditioned on. These are detected and left out of the design rather than silently included, and the exclusion is reported.
A verdict is not a guarantee