Two ideas in.
A number that knows what it doesn't know out.

Error splits in two, and the halves are equal

Take any judgement with a true answer. Bias is the average error — everyone in the room reads high. Noise is the scatter — the same person gives different answers to the same question on different days.

In squared-error terms they simply add:

That equation is why this project exists. The terms contribute equally. Yet organizations spend all their improvement effort on bias — better frameworks, better templates, better training — and none on noise, because noise is invisible in any single decision. You cannot see scatter in one shot. You need a set of judgements of like cases before it appears at all.

Reduce noise and you reduce error even when you have no idea which way your bias runs. That is close to a free lunch, and free lunches are rare.

Three kinds of noise, and only one of them is yours to lose

Noise decomposes further, and the distinction decides what a model can honestly promise.

Level noise is the easy one — one PM is systematically bullish, another is stingy. Stable pattern noise is a judge's idiosyncratic reaction to particular cases: she always over-values developer tooling, consistently, every time. This is usually the largest component, and being stable, it is arguably not error at all — it is her policy. Occasion noise is the same judge, same case, different Tuesday. In one study, developers re-estimating identical tasks differed by an average of 71 percent.

Occasion noise is typically the smallest of the three. And it is the only one a model of your own judgement removes.

Subtraction deletes a whole noise term

Score an idea on its own and you measure the idea plus your level bias plus today's mood:

Score both ideas in the same sitting and subtract, and the shared terms vanish:

A PM who marks everything two points high marks both options two points high. Pairwise comparison is not a gentler kind of scoring — it deletes an entire error term by construction.

Frameworks like RICE do the opposite of this. They multiply and divide four soft estimates, so errors compound rather than cancel — and then the result is printed to one decimal place. 47.3 versus 44.1 reads as information. It is rounding error wearing a suit.

Putting users, weeks and judgements on one scale

Reach is a count of users. Effort is person-weeks. Impact is a −2 to +2 judgement against a named past feature. These cannot be added until they share a scale, so each becomes a z-score:

The μ and σ start as seed distributions shipped in the config, because a new user has no history to standardize against. As the log grows, their own data should take over — and the transition is where it gets subtle. A hard cutover at some threshold changes every subsequent z-score, and so changes verdicts, silently, between sessions. That is precisely the Tuesday-noise the tool exists to remove.

So the default is a Bayesian blend: the seed acts as a pseudo-sample of weight 10 that real data gradually swamps. No discontinuity, and it converges to pure-log.

The whole model is one dot product

Bradley-Terry gives each item a strength and says the probability one beats the other depends only on the difference of strengths, squashed through a logistic function.

Fitting a strength per idea would be useless here — every idea is new and has never been compared to anything. So strength becomes a weighted sum of factors, and the model works on the difference vector:

That is the entire model. Every past decision — “we picked A over B” — becomes one training row: features are the factor deltas, the label is 1. Thurstone's Case V is the same model with a probit link instead of a logistic one, and the choice changes nothing that matters here.

The intercept: forbidden in one role, informative in the other

A symmetric pairwise model must have no intercept. If it had one, the model would believe that whichever option sits in the left column wins more often — a statement about your form layout, not about your ideas. So the scoring fit simply omits the intercept column. That buys an exact guarantee, tested to 1e-12: swap A and B and you get exactly 1 − p.

But that same intercept is interesting. So the code fits it a second time, unconstrained and lightly penalized, purely as a diagnostic. Its z-statistic is a measurement of your presentation-order bias — do you favour whatever you happened to type first? That number appears in the audit view and is never, ever used for scoring.

Why the default weights are not fitted from data

The instinct is to learn the weights. Resist it — the reason is Robyn Dawes' The Robust Beauty of Improper Linear Models.

A proper model optimizes its weights by regression. An improper model uses non-optimal weights — equal weights, or as here, documented prior weights. Dawes showed improper models frequently match or beat properly fitted regression out of sample, and that both beat human intuition.

The mechanism is not mysterious. With small n and correlated predictors, fitted weights chase noise in the training set. Equal weights cannot overfit, because they were never fit. Dawes & Corrigan sharpen it further: nearly all the power of a linear model comes from knowing which variables matter and their sign, not their precise magnitudes.

The flat maximum: stop arguing about decimals

A related and liberating result. When predictors are correlated, the prediction surface is flat across a wide range of weightings. Moving a weight from 0.25 to 0.22 usually does not change the ranking at all.

Which means the open weight questions in this project matter far less than the ordering of the factors. Get the ranking right and defensible; the decimals are configuration, not code, and can be tuned later without touching a line.

A model of you beats you

Lewis Goldberg's Man versus model of man. Build a linear model that predicts the judge's own past judgements. Then set the judge aside and use the model. It wins.

Why? Because it applies the judge's own policy with perfect consistency — it sets occasion noise to zero. In Goldberg's study, judges correlated 0.84 with truth while models of those same judges reached 0.89.

This needs no outcome data whatsoever. No ground truth, no waiting a year to see whether the feature worked. Just a record of what you picked. It activates at ten logged decisions.

Shrink toward the stated policy, not toward zero

This is the piece I would defend hardest, and it is a small idea. Standard ridge regression penalizes weights for being far from zero:

With almost no data that pulls every weight toward zero — toward a model that believes nothing at all. But that is not our belief. Before seeing a single one of your decisions, our belief is the stated prior policy sitting in the config. So the penalty is centred there instead:

What this buys is a behaviour that is correct at both ends and continuous in between. At zero decisions the fit returns exactly the prior, verified to 1e-6. At two hundred decisions the likelihood dominates and you have essentially a free fit. No threshold, no switch, no cliff — the model earns the right to disagree with your stated policy, gradually, in proportion to evidence.

Two fictitious games, or the fit runs off to infinity

A failure mode that bites everyone implementing Bradley-Terry for the first time. If the comparison graph is not strongly connected, or one option wins every single time it appears, the maximum likelihood estimate diverges — the weight sprints to infinity, because infinity fits that data best. With five logged decisions this is not an edge case, it is Tuesday.

The classical fix is pseudo-observations, historically “two fictitious games”. For each factor, the code adds a pair of unit-delta pseudo-comparisons in both presentation orders, with soft outcomes equal to the prior model's own prediction.

Two properties fall out. The fit stays anchored at the stated policy while real data is thin — and because the pair is order-symmetric, it adds no bias to the intercept, so the order-bias diagnostic above stays clean.

A bare probability is a lie of precision

Every verdict carries a 90 percent interval, produced by bootstrapping over the per-factor contributions. The rule that follows is enforced in the code and asserted by a test: the probability is never rendered without its interval. There is no code path in the app that can print “A wins, 59%” on its own. It prints “59% (52–64%)”, or it prints nothing.

Too close to call

If the interval straddles fifty percent, the tool refuses to name a winner. Boundary touching counts as straddling — the tie breaks toward humility.

ζ — a noise meter needing no outcomes and no second person

My favourite instrument in the project, because it measures your own incoherence from your own data, today.

If you prefer A over B, and B over C, and then C over A, you have a circular triad. That is not a mistake about the world — it is an internal contradiction. No ranking whatsoever is consistent with all three of your own choices.

ζ = 1 is perfectly transitive; ζ = 0 is maximally cyclic. It needs no ground truth, no outcomes and no second rater — which makes it the one honesty instrument that works for a PM sitting alone at a desk. The implementation returns None honestly when there is not yet a complete triad to judge, rather than inventing a number.

Being right versus knowing how right you are

Discrimination is ranking — does the tool put better ideas above worse ones? Calibration is honesty about confidence — of all the times it said seventy percent, did the thing happen about seventy percent of the time? They are different virtues and a model can have either without the other.

The mean squared error of a probability forecast; zero is perfect. It accumulates from decision one. A reliability curve — predicted probability binned against observed frequency — unlocks at thirty decisions and not before, because a calibration plot drawn from eight points is itself overconfident. An uncalibrated calibration report would be a fine joke at this product's expense.

Why the c-statistic is not in the product

For a choice model the c-statistic equals the AUC, and it reads beautifully directly: the probability the model ranks a randomly chosen better idea above a worse one. 0.5 is a coin flip. Somers' D = 2c − 1.

Here is why it is absent, and this is the part most tools quietly skip. To distinguish c = 0.80 from chance you need roughly 45 outcome-labeled decisions. For c = 0.70, about 109. If you have five decisions and post “c = 1.000”, you have not measured discrimination — you have measured your sample size.

So the tool ships the metrics that are honest at small n — pick agreement, ζ, Brier — and leaves the outcome scoreboard locked until data exists to support it.

Leverage, exactly — no SHAP required

The verdict pulls apart with no approximation at all:

Each term is factor j's signed contribution to the decision. No attribution heuristic, no sampling, no surrogate model — the model is linear, so the decomposition is the model. It answers “why did it say that”, and it frequently reveals that a verdict you liked rests entirely on a number you guessed. That is the live panel at the top of this page.

Path to parity, in units a person can act on

Given a verdict, what is the smallest single change that flips it? Solve for the shift in factor j that drives the log-odds to zero, then convert back through that factor's standard deviation:

The conversion is the point. Not “raise z by 1.04” but “raise the eval pass rate from 62% to 78%”, or “cut three person-weeks of scope”. The panel computes this for parity and for a 60/40 margin, and checks the target against the factor's plausible range — a path requiring forty million users is arithmetic, not advice, and is marked infeasible.

Lever or fact

Every factor carries a tag. A lever is something you can change by working — eval pass rate, effort, evidence of pull, unit economics. A fact is what it is — problem severity, segment size, capability-trajectory exposure.

Parity paths are proposed only through levers. If every route runs through a fact, the tool says so plainly: this loses on fundamentals; no amount of working harder flips it. That sentence is itself a decision-grade output, and it is the one a spreadsheet will never give you.

Fragility — when a win rests on one guess

The same mathematics pointed the other way. If the winner's margin would flip under a small plausible move of one factor — currently 0.75 standard deviations — the card says so: “A wins, but the margin rests entirely on the effort estimate. Re-estimate before committing.”

A win that survives only if your guess was exactly right is not a win. It is a coin flip with a narrative attached.

Four models, each earning its rung

Each rung replaces the one below only by beating it out of sample on c-statistic and Brier. The equal-weight version stays available throughout as a scientific control.

The direction of travel is the opposite of the industry instinct. More sophistication is not generally better here. With small, noisy, non-stationary data — which is exactly what product decisions are — the simple end of the ladder usually wins out of sample. The ceiling for a real PM tool is low, and that is a feature.

Search and allocation are not the same problem

They share a surface — choose between options under uncertainty — but they sit in opposite statistical regimes, so the tool changes what it claims.

In search mode the banner is permanent, asserted by a test, and the config loader refuses to load the mode without it. The ceiling flag follows the same reasoning: a high-ceiling outlier is routed to human portfolio judgement rather than averaged into a score, because tail-hunting is a portfolio question wearing a ranking question's clothes. Convexity belongs at selection — never inside the probability.

Broken legs, and the honest warning attached

A formula predicts whether someone attends the cinema tonight, and it is a good formula — but she broke her leg this morning. The formula cannot see that. Overruling it is correct.

This is the legitimate case for override, and the app always allows it: pick against the verdict, or against a too-close-to-call, and it is logged with one sentence of rationale. No friction, no nagging.

The catch, and the literature is blunt about it — humans drastically overuse the broken-leg exception. Ordinary cues get treated as decisive, and the override quietly reintroduces the noise the model removed. So overrides are recorded with your stated reason, and can later be audited against outcomes. If your overrides underperform the model — which is the usual finding — that is worth knowing. And you will only know it if you wrote them down.

What the literature actually supports

Meehl (1954) reviewed the comparative studies available then: in all but one, actuarial prediction equalled or beat the clinician. Grove et al. (2000) meta-analyzed 136 studies — mechanical prediction about ten percent more accurate on average, substantially better in 33 to 47 percent of studies, humans substantially better in only 6 to 16 percent.

Read that carefully, because the popular version is wrong. The claim is not that any model beats any human. Models degrade badly with broken-leg cues, tiny samples and non-stationary environments — which describes a great deal of product strategy.