Priors and experiment calibration

A channel prior is a LogNormal on average ROI, so it is fully described by roi_mu and roi_sigma on the log scale. The UI never shows those. It asks for a median and an upper bound and inverts:

mu    = log(median)
sigma = (log(high) - mu) / z

The inverse direction — moment-matching a LogNormal to a target mean and standard deviation — is how the conversions default below is resolved. Experiment calibration borrows only its spread; it keeps a centre of its own, for the reason given with it.

sigma^2 = log1p((sd / mean) ** 2)
mu      = log(mean) - sigma^2 / 2

For a revenue KPI, LogNormal(0.2, 0.9) is used directly. It is already scale-free: a return is a ratio, and nothing about the dataset improves on a median of 1.22× with a 90% band of 0.28× to 5.4×.

For a conversions or sign-ups KPI with no revenue-per-KPI column mapped, “ROI” is conversions per dollar, which has no universal scale — a scale-free prior on it is meaningless. The default is anchored instead on the assumption that media plausibly drives on the order of 40% of the KPI, with deliberately wide uncertainty:

mean = 0.4 * kpi_total / sum(spend)
sd   = 0.2 * kpi_total / sqrt(sum(spend ** 2))

Such a prior is marked scaled_to_data rather than default, because a prior derived from the outcome it will be used to explain is a weaker object than one asserted independently, and the distinction should survive into the artifact.

source · engine/mmm/priors.py · engine/mmm/api/studio.py

A geo experiment or conversion-lift study measures incremental outcome and incremental spend over a test window. Calibration reads that as evidence about the channel’s ROI, a LogNormal centred on the log of what was measured:

roi      = incremental_outcome / incremental_spend
se_roi   = outcome_se / incremental_spend
mu_t     = log(roi)
sigma_t  = sqrt(log1p((se_roi / roi) ** 2))
sigma_t' = k * sigma_t        # k = transportability_discount, default 1.5

It then combines that evidence with the prior the channel had without the test, (mu0, sigma0), by precision on the log scale:

1 / sigma^2 = 1 / sigma0^2 + 1 / sigma_t'^2
mu          = sigma^2 * (mu0 / sigma0^2 + mu_t / sigma_t'^2)

Precisions add, so the result is never wider than the prior or the test. A sharp test all but sets the prior; a weak one barely moves it. Replacing the prior with the test would do worse whenever the test was the wider of the two: the fit would know less with the experiment than without it.

The evidence is centred on log(roi), not on the mean-preserving log(roi) - sigma_t^2 / 2the moment match above would give. Combined, the test is a likelihood of the log return, and a measurement’s likelihood peaks where it read. Its pull on the centre, (sigma^2 / sigma_t'^2) * (mu_t - mu0), fades with its precision, so a test that says almost nothing leaves the prior where it was. The mean-preserving centre would sit lower the weaker the test: one that said nothing at all would still drag mu down by sigma0^2 / (2 * k^2), 0.18 under the default prior and discount.

The discount widens the test’s evidence beyond the experiment’s own standard error, and it is the most important line in this chapter. A lift test measures ROI for the geos, creative and period it ran in. Carrying that number to the whole business at full experimental precision claims more than the experiment established — and because a sharp test all but sets the prior, the claim would propagate into every downstream number with the authority of a randomized result. Widening it never leaves the fit knowing less than it did without the test; it hands weight back to the prior the channel already had. The default of 1.5× is a judgement call and the UI exposes it as one, on a slider, rather than burying it in a config file.

source · engine/mmm/priors.py · roi_prior_from_lift_test · combined_lognormal

Two lift tests on the same channel, run at different spend levels, identify something a single test cannot: the curvature of the response, and therefore the saturation slope. That is route B in the spec, and it needs its own likelihood term.

This engine does not have it. So two tests on one channel are refused — not averaged, not silently reduced to the more recent one. Averaging them would throw away the only information that distinguishes the pair from a single measurement, and would do so invisibly. The error message says which channel and why, and asks the analyst to attach the one they want the prior centred on.