Chapter 05 of 10

Priors and experiment calibration

Eliciting a belief in ROI space, and the one route by which a lift test is allowed to become a prior.

A channel prior is a LogNormal on average ROI, so it is fully described by roi_mu and roi_sigma on the log scale. The UI never shows those. It asks for a median and an upper bound and inverts:

mu    = log(median)
sigma = (log(high) - mu) / z

z is the standard normal quantile for the stated mass; 1.6449 for 90%.

The inverse direction — moment-matching a LogNormal to a target mean and standard deviation — is what experiment calibration uses below.

sigma^2 = log1p((sd / mean) ** 2)
mu      = log(mean) - sigma^2 / 2

Defaults that depend on the KPI

For a revenue KPI, LogNormal(0.2, 0.9) is used directly. It is already scale-free: a return is a ratio, and nothing about the dataset improves on a median of 1.22× with a 90% band of 0.28× to 5.4×.

For a conversions or sign-ups KPI with no revenue-per-KPI column mapped, “ROI” is conversions per dollar, which has no universal scale — a scale-free prior on it is meaningless. The default is anchored instead on the assumption that media plausibly drives on the order of 40% of the KPI, with deliberately wide uncertainty:

mean = 0.4 * kpi_total / sum(spend)
sd   = 0.2 * kpi_total / sqrt(sum(spend ** 2))

Such a prior is marked scaled_to_data rather than default, because a prior derived from the outcome it will be used to explain is a weaker object than one asserted independently, and the distinction should survive into the artifact.

source · engine/mmm/priors.py · engine/mmm/api/studio.py

Route A: an experiment becomes the prior

A geo experiment or conversion-lift study measures incremental outcome and incremental spend over a test window. Calibration is the operation of turning that into the channel’s ROI prior, replacing whatever was there:

roi    = incremental_outcome / incremental_spend
se_roi = outcome_se / incremental_spend
mu, sigma = lognormal_from_moments(roi, se_roi)
sigma = sigma * transportability_discount     # default 1.5

The discount widens the prior beyond the experiment’s own standard error, and it is the most important line in this chapter. A lift test measures ROI for the geos, creative and period it ran in. Carrying that number to the whole business at full experimental precision claims more than the experiment established — and because this prior is strong, the claim would propagate into every downstream number with the authority of a randomized result. The default of 1.5× is a judgement call and the UI exposes it as one, on a slider, rather than burying it in a config file.

Why two experiments are refused

Two lift tests on the same channel, run at different spend levels, identify something a single test cannot: the curvature of the response, and therefore the saturation slope. That is route B in the spec, and it needs its own likelihood term.

This engine does not have it. So two tests on one channel are refused — not averaged, not silently reduced to the more recent one. Averaging them would throw away the only information that distinguishes the pair from a single measurement, and would do so invisibly. The error message says which channel and why, and asks the analyst to attach the one they want the prior centred on.

Calibration is not identification

A calibrated prior is still a prior. If the design cannot identify the channel — flat spend, near-duplicate timing, too few observations — then calibration determines the answer and the posterior will simply return the experiment, contracted slightly. That is often the right outcome, since the experiment is causal evidence and the observational data is not. But it should be described as “the model was told this” rather than “the model found this”, and the learning check in chapter 09 exists to make the difference visible.