Archetype 7

Risk-Factor Weighting CLIOH

turn expert pairwise judgments of predictor variables into a weighted clinical prediction-rule score — an expert prior you can deploy before big-N outcome data exists, then validate empirically later.

InternalCandidateSCORE-AF (→ prediction rule)CORTICES-Distal Tibia (Risk-Factor flavor)pediatric MSKI sepsis-risk score (candidate)

Evidence standard: internal now → external (later) — panel-consensus weights are the expert prior; outcome data is the eventual validating standard
What you compare: Predictor variables (candidate risk factors for a defined outcome)
Panel: Domain experts (clinicians who manage the outcome)
Output: A weighted-score formula (a clinical prediction rule), with the expert weights and their CIs
Validation: Empirical: discrimination (ROC/AUC) and calibration against real outcomes once data accrues

What it's for

A CLIOH study in which experts make pairwise judgments over candidate predictor variables ("which factor raises risk more?"), and the resulting Bradley–Terry worths become the weights of a clinical prediction rule — an expert prior deployable before large-N empirical data, intended for empirical refinement once outcomes accrue.

"In the absence of (or ahead of) a big outcome dataset, what is the best expert-weighted score for predicting this outcome — and how should those weights be revised as real data arrives?"

When to use it

(a) The outcome is rare, slow, or expensive to accrue, so a purely empirical model is years away; (b) experts have strong tacit knowledge of which factors matter; (c) you want a deployable interim score with a principled path to empirical update; (d) the factors are reasonably well-defined and measurable; (e) you intend to validate against outcomes eventually (this is what separates §7 from §1).

When not to

(a) You already have abundant outcome data — fit the empirical model directly (NSQIP-style) and skip the prior; (b) you're ranking variables for understanding, with no prediction-rule or validation intent → that's Discovery §1 (where SCORE-AF/DT live today); (c) you're scoring real cases categorically → Classification §3; (d) you need a real-time operational prioritization, not a risk score → Triage §4 (though see H4 — one elicitation can yield both); (e) the outcome is so ill-defined that the prediction target is meaningless.

What you get

A weighted-score formula (the clinical prediction rule) with the expert weights and their confidence intervals, plus a stated plan and schema for empirical validation/update. Not a deployed clinical tool until validated against outcomes + IRB-reviewed; the UI carries no unverified performance or regulatory claim.

A real example

SCORE-AF (arthrofibrosis risk factors) — Discovery §1 today; turned into a weighted arthrofibrosis-risk prediction rule with outcome validation = Risk-Factor CLIOH.
CORTICES-Distal Tibia (growth-arrest predictors) — Discovery §1 with Risk-Factor flavor; → prediction rule for growth-arrest risk, or → Classification §3 if applied to real growth-arrest cases with outcome linkage.
Pediatric MSKI sepsis-risk score (candidate) — the DR's worked example; expert-weighted predictors for musculoskeletal-infection sepsis risk, primed before a large prospective cohort exists.
Combined with live case scoring, this produces Hybrid H4 (Triage × Risk-Factor): one covariate-structured BT elicitation yields both a real-time triage score and the implicit risk-factor weights as a second deliverable.

Validation

Internal first (test-retest of the elicitation, panel-composition sensitivity). Then empirical: discrimination and calibration against real outcomes as data accrues; head-to-head against any incumbent score; recalibration on update. Pre-register the validation plan so the expert-prior and the empirical-test are independent. Report the transition explicitly — "expert prior" → "expert prior, empirically validated."

Common pitfalls

(a) Confusing causal with predictive — experts may rank a factor high because it's causally important even if it adds little predictive value; define the elicited construct as predictive contribution. (b) Treating the expert prior as validated — it isn't until outcomes confirm it. (c) Collinearity among predictors distorting worths. (d) The prior never updating because the validation cohort never gets built — commit to the data plan up front. (e) Strength-Spread failure — no natural anchors, so a slate without deliberate HIGH/MED/LOW design produces a compressed, uninformative weight scale. (f) Circular validation — expert exposure to outcome data before elicitation. (g) Confusing this with Discovery — if there's no prediction-rule or validation intent, you're still in §1.

How it works — show me the method

Item selection. Candidate predictor variables, selected for clinical plausibility and measurability. Strength-Spread §1.1 binds hard here — predictor variables have no natural scale endpoints, so the slate must deliberately include HIGH-weight contenders, MEDIUM bridges, and LOW anchors, with anchors pre-tested at Stage 0 Gate A. (This is one of the three archetypes — with Discovery and Triage-criteria — where the absence of natural anchors makes Strength-Spread most load-bearing.)

What you compare. Each item is a predictor, and the worth being elicited is its contribution to the outcome, not its prevalence or its severity-as-experienced. Define the outcome precisely before elicitation. Watch for collinearity — two predictors that travel together will split or distort each other's worths; flag known correlated pairs for the analyst.

Comparison structure. Bare-vote, binary forced-choice pairwise "which factor contributes more to [outcome]?", NO ties (ADR-CLIOH-03). BIBD pair allocation; ATRD Round 2 (TRE) on contested pairs (ADR-CLIOH-04). Where the prediction rule will operate on cases with covariates, covariate-structured BT (Springall 1973; Cattelan 2012) links the score to case features and is the bridge to Triage §4 (see H4). Single-step back button (ADR-CLIOH-02).

Panel. Domain experts who actually manage the outcome — 15–30 (target 25–30; if N<15, Cooke-weighted aggregation rather than hierarchical Bayes). Breadth across practice settings improves the prior's generalizability before empirical data exists to correct it.

External ground truth. Risk-Factor is internal at elicitation (the panel consensus is the prior) but external on validation — you must eventually link the score to real outcomes (ROC/AUC, calibration). Until you do, the claim is "expert-weighted," not "empirically validated"; say so. Circular-validation hazard: don't let any outcome data the experts have seen leak into the elicitation, or the prior and the validation aren't independent.

Statistical backbone. Frequentist BT MLE via choix (bootstrap CIs, Firth) yields the weights. The natural next step is to treat the BT worths as informative priors in a subsequent logistic / Bayesian prediction model fit on accruing outcome data — the expert prior is updated toward the empirical posterior as N grows (the prior dominates when data is thin and recedes as data accumulates). This prior-then-update structure is where Bayesian-hierarchical BT is genuinely warranted rather than conditional (ADR-CLIOH-07 draft; Playbook §7). The empirical comparator is the ACS-NSQIP paradigm: a calculator built from 1.4 million patients across 393 hospitals on 21 preoperative factors (Bilimoria et al. 2013, PMID 24055383) — CLIOH-Risk-Factor primes exactly that kind of model with expert weights when the 1.4M patients don't yet exist.

For researchers — reporting, IRB & grant language

Reporting standards. TRIPOD / TRIPOD-AI (prediction-model development and validation) is the primary standard; report the model as development with expert priors and flag validation status honestly. SQUIRE if framed as QI. LitGuard every citation; DLRP Zotero mirror.

IRB / ethics. Human-subjects on the validation side (outcome data); the elicitation itself is expert-opinion (often QI/exempt) but make the QI-vs-research determination up front. No PHI in the elicitation instrument; aggregate-only display. Do not present an unvalidated expert-prior score as a validated risk calculator — that's both an ethics and a regulatory exposure. No unverified IRB claim in the UI.

Grant-application language.

"The ACS-NSQIP Universal Surgical Risk Calculator was built from 1,414,006 patients across 393 hospitals using 21 preoperative factors (Bilimoria et al., J Am Coll Surg 2013, PMID 24055383) — a definitive example of empirical risk-weight estimation, but one that required a national registry and years of accrual. For rare pediatric-orthopaedic outcomes where such datasets do not yet exist, Risk-Factor Weighting CLIOH elicits expert pairwise judgments of candidate predictors and fits a Bradley–Terry model to derive interval-scale weights, producing a deployable clinical prediction rule that functions as an informative Bayesian prior — refined toward the empirical posterior as outcome data accrues. The approach builds on validated pairwise-comparison precedent in which Bradley–Terry severity parameters outperformed conventional ordinal scales (Pedersen et al., BMJ Open 2021, PMID 34006025, knee-OA AUC 0.97 vs 0.80–0.88), and supplies the expert-derived starting weights that large-N registry models like NSQIP otherwise require years of data to learn."