Methods primer
How CLIOH works
Two depths. The gist tells you what we are doing and why; the equations tell you which likelihood we are fitting. Both arrive at the same place: an interval-scaled ranking with bootstrapped 95% confidence intervals.
The gist
People are good at comparing two things and bad at rating one thing on a numeric scale. Likert scales ask experts to do the hard thing; CLIOH asks them to do the easy thing. The bare-vote forced-choice format — "A or B, which is more severe?" — strips away the false precision of 1-to-10 ratings and harvests the one signal experts actually have: ordinal preference.
Each item ends up with a worth on a continuous interval scale. The differences between worths are meaningful: the gap between rank 1 and rank 2 might be larger or smaller than the gap between rank 3 and rank 4, and the model tells you which. That is what "interval-scaled" means and why it matters: a Likert-derived ranking gives you order; a Bradley–Terry-derived ranking gives you magnitude.
The 95% confidence intervals come from bootstrap resampling — re-running the fit on resampled votes thousands of times and reporting the central 95% of the resulting worth distribution. Where the CIs overlap, the items are statistically indistinguishable; where they separate, the ranking is real.
The three layers
1. Thurstone
Thurstone's 1927 Law of Comparative Judgment posits that each item has a latent quality drawn from a normal distribution, and the probability that item A beats item B in a paired comparison is the area where A's draw exceeds B's. This is the conceptual foundation: a measurable, interval-scaled latent variable recovered from pairwise judgments.
2. Bradley–Terry
Bradley and Terry (1952) replaced the normal with a logistic, yielding a likelihood that is easier to fit and identical in form to logistic regression. Each item gets a worth π; the probability item i beats item j is πi / (πi + πj), equivalently a logistic of the log-worth difference. This is the workhorse — fast, well-studied, with closed-form Fisher information. We fit it via maximum likelihood (choix in Python) with bootstrap CIs and a Firth penalty when separation threatens.
3. Conditional hierarchical Bayes
When the panel is small, the items are many, or we need subgroup analysis (site-by- site, attending-by-resident), hierarchical Bayesian Bradley–Terry shrinks toward the panel mean and yields full posteriors instead of point estimates plus CIs. The packages we use are bpcs (R) and PyMC. This is the conditional path — not the default — because the frequentist BT is usually enough and cheaper to compute.
Why this is the same likelihood AI reward models use
Reinforcement Learning from Human Feedback (RLHF) trains preference reward models from pairwise human judgments. The likelihood under the hood is Bradley–Terry. CLIOH is doing the same thing — fitting a BT preference model — except the panel is a board- certified clinician cohort and the items are clinical variables, real cases, or case features instead of language-model completions.
The methodological lineage matters: BT-from-pairwise is not novel; it is the standard tool in two fields that have spent serious money de-risking it — psychometrics and machine learning. Bringing it to clinical consensus is not invention, it is borrowing.
The locked rules (Shared Core)
Every CLIOH study, regardless of archetype, follows the same operational core:
- Bare-vote, binary forced-choice, no ties. The model handles "these two look equally bad" via a win probability near 0.5; reaching for a tie option corrupts the BT likelihood.
- BIBD pair allocation. Balanced incomplete block designs ensure each item appears in a fixed number of comparisons (r ≈ 8) and each pair appears equally often.
- ATRD on contested pairs. Round 2 is an asynchronous targeted re-vote on pairs where the panel split most evenly — a way to extract dissent rather than smother it.
- Strength-Spread. The item slate must span the strength range (anchors at the bottom, contenders at the top). All-contender slates produce overlapping CIs and uninterpretable rankings.
- Unanimity Gate. Whether respondents are shown anonymized or named to each other depends on whether unanimity is reached on high-impact items.