Archetype 9

AI / Algorithm Calibration CLIOH

label or benchmark a clinical AI's outputs against an expert-consensus interval scale, so the model is measured on the surgeons' own ruler — not an arbitrary one.

ExternalCandidateMeridian (machinery / LLM-as-panelist)CORTICES case-bank AI benchmark (candidate)

Evidence standard: external (hybrid — panel = proximal standard; outcome dataset = distal standard)
What you compare: Model outputs on real cases (the model's score/label vs the panel's worth)
Panel: Domain experts (surgeons) + the ML/data-science team
Output: A calibrated benchmark: model score ↔ expert worth (with agreement metrics + a leaderboard if multiple models)
Validation: Concordance of model output with the BT worth scale; ROC/AUC against the distal outcome layer; drift monitoring

What it's for

A CLIOH study that produces an expert-consensus interval scale and then uses it to calibrate, benchmark, or label a clinical AI model — either by training the model toward the BT worths (regression target) or by scoring competing models against them (leaderboard). The panel anchors the model.

"Does the algorithm agree with the surgeons — on the surgeons' own measured scale — and which of several models agrees best?"

When to use it

(a) A clinical AI / SaMD candidate exists or is in training and needs a defensible benchmark; (b) the relevant construct (severity, urgency, plus-disease, fracture grade) has no clean numeric gold standard, so an expert-consensus scale is the best available reference; (c) you want continuous regression targets for model training rather than coarse ordinal labels; (d) you are comparing multiple models and need a principled ranking; (e) regulatory / publication pressure demands the benchmark be reproducible and tied to expert judgment.

When not to

(a) A hard outcome label already exists and is cheap (use it directly — don't proxy through experts); (b) you are ranking variables, not scoring model outputs → Discovery §1; (c) you are grading real cases categorically with no model involved → Classification §3; (d) you want to evaluate a human trainee's judgment, not a model → Calibration §5; (e) the model is being deployed to make decisions in real time → that is a Triage §4 deployment problem with its own SaMD obligations layered on top.

What you get

A calibration report: the expert worth scale, the model↔worth mapping with confidence bands, agreement metrics at both truth layers, and — for multi-model studies — a Bradley–Terry leaderboard. This is a benchmark, not a deployed clinical decision tool. Any move to deployment requires separate IRB review, prospective validation, and an SaMD-grade quality management system (binding caveat carried from Triage §11). The instrument UI makes no unverified regulatory claim.

A real example

CORTICES case-bank AI benchmark (candidate first instance): three radiograph fracture-detection / -grading models scored on a Strength-Spread CORTICES case bank; experts render pairwise worths; models ranked by concordance with the worth scale on a BT leaderboard. The reusable artifact is the calibrated case bank itself.
Meridian (machinery): the AI-consensus / persona layer and the LLM-as-panelist module are the lab's vehicle for §9. LLM-as-panelist — a calibrated medical LLM standing in as a virtual panelist — is genuinely novel (no pediatric-ortho group has done it) and is treated as research methodology, not deployment; validate the LLM against humans on calibration items first. Robust-reward-modeling work flags real failure modes — reward-hacking and overconfidence (Liu, Ge & Zhu 2024, arXiv:2410.05328; PURM, arXiv:2503.22480) — which must be designed against.
i-ROP precedent (Classification §3 → §9): the i-ROP pipeline (PMID 35157950) is the cleanest existing clinical instance of this archetype and is exactly Hybrid H3 (Classification × AI-Calibration) — the BT/Elo expert worths become continuous regression targets for the deep-learning model.

Validation

Concordance of model output with the expert worth scale; ROC/AUC of both against the distal outcome dataset; head-to-head leaderboard stability under bootstrap; drift monitoring (dynamic BT) for production models; panel-composition sensitivity (drop-one-out) so the benchmark isn't an artifact of which experts were recruited. Pre-register the benchmark protocol.

Common pitfalls

(a) Letting the proximal layer pose as ground truth — the expert scale is a standard, not the truth; always report the distal outcome layer too. (b) Circular validation — experts must not see model output or outcomes while rating. (c) Incommensurable targets — defining the model↔worth mapping after elicitation instead of before. (d) Case-mix / Strength-Spread failure — a case bank that misses the borderline range yields a benchmark that only looks good. (e) Reward-hacking / overconfidence in the LLM-as-panelist variant — treat as a known failure mode, not an edge case. (f) Drift — a model benchmarked once and never re-checked (use dynamic BT). (g) Over-claiming regulatory status in the UI — it's a benchmark until separately validated as SaMD.

How it works — show me the method

Item selection. Items are real cases run through the model (radiographs, scenarios, structured profiles). The model emits a score/label per case; the panel renders pairwise worths over the same cases. The case bank must satisfy Strength-Spread §1.1 — span clearly-high, borderline, and clearly-low cases — or the calibration curve is estimated only over part of the range and the benchmark over-claims. Anchor cases pre-tested at Stage 0 Gate A.

What you compare. The construct must be the same one the model targets (e.g., "plus-disease severity," "fracture displacement," "operative urgency"). The expert worth scale and the model output must be commensurable — if the model predicts a 5-class label and the panel measures a continuous worth, define the mapping before elicitation, not after.

Comparison structure. Bare-vote, binary forced-choice pairwise "which case is more [construct]?", NO ties (ADR-CLIOH-03 — the DR's Davidson-ties recommendation is rejected here as everywhere). BIBD pair allocation; ATRD Round 2 (TRE) on contested pairs (ADR-CLIOH-04). Where cases carry covariates relevant to model behavior, covariate-structured BT (Springall 1973; Cattelan 2012) links worth to features. For longitudinal benchmarking of a model in production, dynamic / time-varying BT (Cattelan; Varin & Firth 2013) monitors whether model–expert agreement drifts as practice or the model changes. (Active-learning / entropy-driven pair selection = future enhancement, not in the locked core.)

Panel. Two constituencies: domain experts (the surgeons whose judgment is the proximal standard) and the ML/data-science team (who own the model and the target definition). 15–30 experts for the worth scale. Including the ML team early prevents the classic failure where a beautiful expert scale is incommensurable with what the model actually emits.

External ground truth. This is the field's subtlety and the reason AI-Calibration is logged as hybrid rather than purely external. Proximal standard: the expert-consensus BT worth scale (the panel is the immediate reference the model is scored against). Distal standard: a true outcome dataset (mortality, progression, complications, biopsy/path label) against which both the model and the expert scale can be checked. Report agreement at both layers; do not let the proximal layer masquerade as ground truth. Circular-validation hazard: if the experts saw model output (or outcome data) while rating, the benchmark is contaminated — blind the elicitation.

Statistical backbone. Frequentist BT MLE via choix (bootstrap CIs, Firth penalty for separation) produces the worth scale. Model calibration assessed by concordance of model score with worth (concordance correlation, Spearman, calibration plots) and by ROC/AUC against the distal outcome layer. When multiple models compete, fit a Bradley–Terry / Elo leaderboard over model-vs-model agreement — the same machinery Chatbot Arena and MedArena use. Bayesian-hierarchical BT is conditional (small case banks, subgroup calibration), not default (ADR-CLIOH-07 draft; Playbook §7).

For researchers — reporting, IRB & grant language

Reporting standards. TRIPOD-AI (model development/validation) + DECIDE-AI (early clinical evaluation) where applicable; CLAIM for imaging-AI; SQUIRE if framed as QI. LitGuard every citation; DLRP Zotero mirror. State the QI-vs-research determination up front (changes the IRB path).

IRB / ethics. Human-subjects if cases are patient-derived; QI-vs-research determination up front. No PHI in the elicitation instrument; aggregate-only display. The model's training-data provenance and any patient-data governance are separate, heavier obligations from the benchmark itself. No unverified IRB/IACUC or FDA/SaMD claim in the UI (Hard Rules #7, #11–12).

Grant-application language.

"Reinforcement Learning from Human Feedback — the training procedure behind every frontier large language model — fits a reward model whose likelihood, P(i ≻ j) = σ(rᵢ − rⱼ), is identical to the Bradley–Terry likelihood CLIOH fits to expert clinical judgment (Christiano et al., NeurIPS 2017; Lambert, The RLHF Book 2024). The same machinery now ranks large language models for the public (Chatbot Arena: 1.37 million pairwise human votes over 90+ models — Chiang et al., ICML 2024, arXiv:2403.04132) and for clinicians specifically (MedArena: 1,571 clinician preferences across 12 models — Wu et al., Stanford Biomedical Data Science, 2025, arXiv:2603.15677). In medical imaging, the i-ROP consortium used pairwise expert comparisons to validate a deep-learning vascular severity score against International Classification of Retinopathy of Prematurity expert labels, achieving a vascular-severity-score correlation of 0.90 (Ophthalmology 2022, PMID 35157950). AI-Calibration CLIOH brings this paradigm to pediatric orthopaedic surgical AI: an expert-consensus Bradley–Terry worth scale serves as the proximal benchmark against which candidate models are calibrated and ranked, with outcome data as the distal standard — producing a defensible, reproducible, expert-anchored benchmark for FDA-pathway surgical AI."