BABTP
Bayesian A/B Testing Primer
Six sections, one idea. A conversion rate is a probability you do not know, so you represent it with a distribution rather than a number. Everything that follows — posterior updating, the chance one variant beats another, the cost of deciding early — is that one move applied again and again.
Section 1 · The Beta distribution
A distribution over a probability
The Beta distribution is the standard prior and posterior for a conversion probability: flexible, bounded between 0 and 1, and it updates cleanly after observing successes and failures. Its density is
where p is a probability between 0 and 1, and α and β are the shape parameters. The normalizing term B(α, β) is the Beta function — the constant that makes the area under the curve exactly 1, so the curve is a valid distribution.
A larger α pulls belief toward higher conversion probabilities, a larger β toward lower ones. When both are 1 the distribution is flat, which is why Beta(1, 1) is the usual neutral starting point.
The current prior is Beta(1.00, 1.00), with a mean of 50.0%. The shaded region is the whole density, and its area is always 1 however you move the parameters.
Why this matters for A/B testing
A conversion rate is a probability, so we want a distribution defined on probabilities. The Beta family gives a compact language for prior belief and an easy update rule once data arrives. It is also conjugate to the binomial likelihood, which means the posterior stays Beta after you observe data:
Section 2 · From shape to conversion data
Counting into the shape
In an A/B test, α and β stop feeling abstract as soon as you connect them to outcomes. Start from a neutral Beta(1, 1), then add successes to α and failures to β.
As the sample grows the curve narrows, because the posterior becomes more certain. To make the squeezing obvious, hold the conversion rate fixed at 5% and compare three sample sizes.
| Sample | Successes | Failures | Alpha | Beta |
|---|---|---|---|---|
| N = 100 | 5 | 95 | 6 | 96 |
| N = 500 | 25 | 475 | 26 | 476 |
| N = 2,000 | 100 | 1,900 | 101 | 1,901 |
Three posteriors from the same 5% observed rate and a Beta(1, 1) prior. At N = 100 the posterior is still broad; by N = 2,000 it has squeezed tightly around 5%, which is what it means for data to be informative about the underlying rate.
Section 3 · Comparing two posteriors
Which variant is better?
Each variant gets its own stream of data, so each gets its own posterior. By the time you read results you have two Beta distributions side by side, and the question is no longer what the rate is — it is which variant is actually better.
For every value the variant’s rate might take, ask how much of the control’s posterior falls below it. Integrating across the range gives the probability that the variant’s true rate exceeds the control’s:
That integral has no closed form for arbitrary parameters, so both posteriors are evaluated on a fine grid. Stepping left to right, two running totals are maintained at once:
The first line builds up the control’s CDF incrementally. The second weights the variant density at each point by how much of the control’s distribution lies below it. Both posteriors are smooth and well concentrated, so the approximation converges quickly.
P(B > A)91.0%tick marks 95%
Control dashed, variant solid; the shaded region is where they overlap. That overlap is the zone of uncertainty — both rates are plausible there. When the curves are far apart P(B > A) approaches 0 or 1 and the decision is clear. When they nearly coincide it drifts toward 50%, which does not mean the test is broken: it means the data cannot yet tell the variants apart.
Why a probability beats a verdict
The same 2-point lift at N = 5,000 produces a far more decisive result than at N = 100, because more data compresses both posteriors without moving their centres.
Instead of a binary reject-or-not decision, you get a continuous probability that can be monitored, communicated, and acted on as evidence accumulates.
Section 4 · Expected loss
The cost of being wrong
P(B > A) gives the direction but not whether acting on it is wise. A 90% probability sounds decisive, but if B is only fractionally better, the cost of being wrong and the cost of being right are nearly the same. The question that drives the decision is: how much conversion rate do I sacrifice on average if I choose wrong?
LossB is the average conversion rate you give up by shipping B if A was secretly better; the max(·, 0) counts only the cases where A genuinely wins. LossA is the symmetric regret of holding B back when it was stronger. Ship whichever variant has the lower cost of being wrong about it.
| Hold A — cost of withholding B | 2.058 pp |
| Ship B — cost of shipping early | 0.066 pp |
Expected loss for each decision at the current rates and sample size. The rule is to ship B when LossB is below LossA, which here says ship B. Both numbers come from the same left-to-right pass over the grid that produced P(B > A).
Section 5 · Should you keep running?
What another week buys
Expected loss answers whether to act on the data you have. A different question is whether collecting more is worth it. If LossB is already near zero, another week changes almost nothing. If both losses are still high and close together, more data has real value because it might separate them.
Holding the observed rates constant, you can compute what the posterior — and therefore the expected loss — would look like at any future N. Both losses shrink as N grows, but not at the same rate: when one variant is genuinely better, its loss converges to zero much faster than the other.
Both losses projected forward from the current sample, in percentage points. Today they stand at 2.260 for holding A and 0.280 for shipping B; by N = 1,200 per arm, shipping B costs 0.007. Where one line drops to near zero while the other stays high, the decision is settled.
Protecting against noise
Comparing LossB to LossA gives the direction. It does not protect against acting too early: in a noisy test LossB can be the smaller number on a small sample and reverse a week later, because both posteriors are still wide and the comparison is fragile. The fix is a pre-committed absolute threshold.
A principled starting point is the minimum lift your business would actually act on: if 0.5 percentage points is the smallest effect worth shipping, set ε = 0.005. The threshold requires the magnitude of the loss to fall below a meaningful level, not merely below the other loss. The sample size you need is the point where the projected LossB curve first drops under it:
The full stopping rule is two conditions at once:
- LossB < ε — the expected regret of shipping is below your threshold.
- N ≥ Nmin — you have enough data for the projection to be credible at the observed rates.
That rule would have blocked the premature calls in the walkthrough below.
Section 6 · A test, week by week
The recommendation flips twice
New users arrive each week and the posteriors update. Below is the whole run at once — one small chart per week, control dashed and variant solid, on a shared reading. Select a week to read what was happening at the time.
| Week | N / arm | P(B > A) | Loss A | Loss B | Call |
|---|---|---|---|---|---|
| 1 | 250 | 99.2% | 5.165 | 0.006 | Ship B |
| 2 | 500 | 79.2% | 1.380 | 0.184 | Ship B |
| 3 | 750 | 37.9% | 0.315 | 0.714 | Hold A |
| 4 | 1,000 | 39.5% | 0.287 | 0.586 | Hold A |
| 5 | 1,250 | 63.9% | 0.558 | 0.238 | Ship B |
| 6 | 1,500 | 76.3% | 0.723 | 0.123 | Ship B |
Losses are in percentage points. The call column is simply whichever loss is smaller — and it flips at week 3 and again at week 5. In weeks 1 and 2 both losses are high in absolute terms even though LossB is the smaller one: the signature of a wide posterior, where the comparison points in a direction but the magnitude says the data is thin.
Week 1 — B jumps out early
Only 250 users per side and B is already sitting well to the right of A's posterior. P(B > A) is high and Loss_B is the smaller loss. The math favors shipping B — but the posteriors are still very wide. Notice how elevated both losses are in absolute terms. This is a signal that the data is thin, not that B is proven.
| N per variant | 250 |
| P(B > A) | 99.2% |
| LossA / LossB | 5.165 / 0.006 pp |
| Recommendation | Ship variant B |
What this walkthrough shows
B looked like the clear winner in weeks 1 and 2, then A pulled ahead for weeks 3 and 4, then B reclaimed the lead. The true rates — 5.0% for the control and 6.5% for the variant — were never visible during the test. The early lead for B was variance; A had two strong weeks; then the true signal reasserted itself.
A pre-set threshold ε would have held the test through those early weeks regardless of which loss was lower. That is the full loop: prior, posterior update, P(B > A) for direction, expected loss for stakes, the threshold and Nmin for noise, and the projection for deciding whether to wait. Every piece is the Beta distribution from section 1, used again.