Frequently asked questions

No. Classic Delphi asks experts to rate items on a numeric scale and then re-rate after seeing the group's rating distribution. That is the procedure CLIOH was built to replace, for two reasons:

• Likert ratings are not interval-scaled. The gap between "6" and "7" might mean something different to each expert and is not comparable across items.
• Re-rating after seeing the group rating creates bandwagon dynamics rather than recovering the panel's independent signal.

CLIOH replaces the Likert step with pairwise comparisons (the easy thing to ask experts to do), fits Bradley–Terry to recover an interval-scaled worth per item, and uses targeted re-voting (ATRD) on contested pairs rather than universal re-rating.

People reliably know which of two things is more X — that is the comparison they make every day in clinical practice. People do not reliably know whether something is a 7 or an 8 on an unanchored scale. Pairwise harvests the signal experts actually have; Likert manufactures a signal they have to invent. Fit a Bradley–Terry model to the pairwise data and you get back what the Likert was trying to produce — except now it is interval-scaled and the confidence intervals are honest.

For a Discovery archetype with k = 13–19 items, the Methodology Playbook calls for a minimum of 15 experts and a target of 25–30, stratified by site and years-in-practice to enable subgroup analysis. Smaller panels are recoverable with hierarchical Bayesian BT (the conditional path), but the frequentist primary analysis benefits from N ≥ 25.

The locked rule is bare-vote, binary forced-choice, no ties (ADR-CLIOH-03). It is counterintuitive — the obvious move would be to add a "they're the same" option — but adding ties corrupts the Bradley–Terry likelihood and produces a model that fits less well, not more.

The BT model already handles "these two look equally bad": it assigns a win probability near 0.5, the bootstrap CIs widen, and the two items end up adjacent on the worth scale with overlapping CIs. The information is preserved without the model penalty.

It uses the identical likelihood. RLHF reward models are Bradley–Terry models fit to human pairwise preferences over language-model completions. CLIOH is a Bradley–Terry model fit to clinician pairwise preferences over clinical items. The math is the same; the application domain and the panel are different. That methodological convergence is a feature: BT-from-pairwise is the standard tool in two fields (psychometrics and machine learning) that have spent serious money de-risking it.

Yes — that is Archetype 8 (Patient-Centered Preference) or a mixed-panel hybrid. The spec sheet for Archetype 8 is queued; the design considerations differ from expert-only archetypes (different anchoring, different Strength-Spread expectations, more attention to literacy / accessibility of item descriptions). Submit a project and we will route it once Archetype 8 is written.

Research-only, for now. There is methodological precedent — Christiano et al. 2017 established that LLMs can produce BT-stable pairwise preferences — but using an LLM as a clinical-consensus panelist conflates "what the model thinks" with "what clinicians think," and the goal of CLIOH is the latter. The right use is methodological: have the LLM act as a synthetic panelist alongside human experts for power calculations, pilot testing, or anchor-spread validation.

No. This is a binding constraint of the methodology. The Triage CLIOH (Archetype 4) produces an interval urgency score that maps case features to a recommended order, but it is not a clinical decision tool until separately:

• IRB-reviewed for deployment
• Prospectively validated against patient outcomes at the deployment site
• SaMD-grade quality-management-system instrumented

The methodology page produces the score; the deployment phase is a separate clinical engineering effort and a heavier regulatory path.

Bradley–Terry pairwise models for ordinal grading have been the dominant approach in ophthalmology — the i-ROP vascular severity scale (Campbell et al., Ophthalmology 2022, PMID 35157950) is the methodological cousin most relevant to clinical CLIOH. Adjacent work in educational comparative judgment (Pollitt, Verhavert) and AI preference learning (Christiano et al.) uses the same likelihood for the same reason. CLIOH packages the locked operational rules, the Strength-Spread requirement, and the archetype taxonomy on top of that statistical backbone.

The five locked rules (bare-vote / BIBD allocation / ATRD round 2 / Strength-Spread / Unanimity Gate) are summarized on the methods primer and detailed on each archetype spec sheet under "How it works." The consortium's methodology playbook (CORTICES-internal) contains the full operational specs.