Archetype 5
Calibration / Assessment CLIOH
measure how well an individual's clinical judgment agrees with an expert-consensus reference scale — i.e., score a person, not a variable.
- Evidence standard
- internal→external (two-layer — faculty consensus = proximal standard; outcome dataset = distal standard)
- What you compare
- Standardized vignettes / cases (the item set is fixed; the examinee varies)
- Panel
- Two roles: faculty (build the reference scale) + examinee learners (scored against it)
- Output
- An individual calibration report (agreement with the reference scale, with CI), optionally a learning curve over time
- Validation
- Test-retest of the reference scale; criterion checks vs the distal outcome layer; sensitivity to faculty-panel composition
What it's for
A CLIOH study that first builds an expert-consensus interval scale over standardized cases ("the answer key"), then has individuals make the same pairwise judgments and scores each individual by how closely their implied worths agree with the reference. It is structured expert-judgment calibration (Cooke's seed-question logic) re-expressed in Bradley–Terry terms.
"How closely does this resident / fellow / re-certifying surgeon's judgment match the faculty consensus — and is the gap closing over training?"
When to use it
(a) A defensible reference scale already exists or can be built from faculty pairwise judgments over standardized items; (b) you need to assess judgment (ranking/prioritization sense), not factual recall; (c) program-level use — residency milestones, fellowship readiness, MOC/re-certification, OITE-style assessment; (d) you want a continuous, calibrated score rather than a coarse entrustment level; (e) longitudinal tracking of a cohort is feasible (learning curves).
When not to
(a) You are building the faculty scale itself, not scoring anyone against it → that's Discovery §1 / Educational §2; (b) a validated factual exam already covers the competency (don't proxy recall through pairwise judgment); (c) you are scoring an algorithm, not a person → AI-Calibration §9; (d) the construct has a clean external gold standard the examinee could be tested against directly (use it); (e) the stakes are high-consequence certification and the reference scale lacks validity evidence — do not make consequential pass/fail calls off a normative scale that hasn't been validated against outcomes (see §14).
What you get
An individual calibration report: the examinee's agreement with the faculty scale, a confidence band, where they diverge most (which item types), and — for tracked cohorts — a learning curve. Program-level: a class distribution and norm. The instrument makes no high-stakes certification claim it can't support (see §14); aggregate-only public display.
A real example
- Resident calibration on CORTICES OR/ER-Priority (candidate first instance): the attending faculty scale already exists as a Discovery §1 instrument; residents complete the same standardized pair set; each resident's agreement with the attending worth scale is their calibration score. The reusable artifact is the calibrated case bank + answer key.
- H1 Calibration × Educational (the high-yield combination): build the faculty norm (Educational §2), then deploy a calibration quiz against trainees (§5). Highest-yield for residency programs — one elicitation produces both a teaching tool and an assessment. Target institutionalization for the AAOS OITE.
- Dynamic learning curves: track a residency cohort across years with time-varying BT → an individual learning curve that is more sensitive than discrete EPA milestone steps.
Validation
Test-retest of the reference scale; faculty-panel-composition sensitivity (drop-one-out — the answer key must not hinge on which faculty showed up); criterion validity against the distal outcome layer where it exists; predictive checks (do higher-calibration trainees have better downstream outcomes?). Pre-register the scoring rule before collecting examinee data. Threshold: if an LLM is ever used as a virtual examinee/panelist and its agreement κ<0.5 on calibration items, do not deploy it for that domain (DR threshold).
Common pitfalls
(a) A noisy reference scale — if the faculty answer key isn't test-retest stable, no calibration score means anything; validate the key first. (b) Conflating normative with predictive — "agrees with faculty" ≠ "makes better decisions" unless the distal outcome layer says so. (c) Encoding faculty bias as the standard and then penalizing learners who diverge correctly. (d) High-stakes use without validity evidence — do not gate careers on an unvalidated normative scale. (e) Letting examinee data leak into the reference. (f) Case-mix / Strength-Spread failure — an item bank missing the borderline range produces a calibration score that can't discriminate. (g) Treating it as factual testing — this measures judgment, not recall.
How it works — show me the method
Item selection. A fixed bank of standardized vignettes / cases spanning the construct, satisfying Strength-Spread §1.1 (clearly-high, borderline, clearly-low) so the calibration score discriminates across the whole range — not just at the easy ends. Anchor items pre-tested at Stage 0 Gate A. The item set is held constant; the examinee is the variable. A subset functions as seed/calibration items (Cooke's "questions of known truth") — here, "known truth" = the faculty-consensus worth.
What you compare. The examinee judges the same construct the faculty scale measures (urgency, severity, operative priority, plus-disease grade). The reference scale and the examinee's judgments must be commensurable — both are pairwise worths over the same items, which is what makes the agreement metric clean.
Comparison structure. Bare-vote, binary forced-choice pairwise, NO ties (ADR-CLIOH-03 — the DR's Davidson-ties recommendation is rejected). BIBD pair allocation; ATRD Round 2 (TRE) used when building the faculty reference (ADR-CLIOH-04). For the examinee pass, the individual completes the standardized pair set; their implied worths are compared to the reference. Longitudinal → dynamic / time-varying BT (Cattelan; Varin & Firth 2013) to produce an individual learning curve across residency — more sensitive than discrete EPA milestone levels. Single-step back button on the survey (ADR-CLIOH-02).
Panel. Two distinct roles. Faculty (reference builders): 15–30 senior experts whose consensus is the proximal standard (target 25–30 per Playbook; if N<15, switch to Cooke-weighted aggregation rather than hierarchical Bayes — DR threshold, adopted). Examinees: the learners being scored — residents, fellows, MOC candidates. Keep the two passes cleanly separated; an examinee's data never feeds back into the reference.
External ground truth. Proximal: the faculty-consensus scale (the immediate answer key). Distal (where available): real outcome data, against which both the faculty scale and high-scoring examinees can be checked — this is what lets the program claim the calibration score predicts better decisions, not just conformity. Without the distal layer, the claim is normative ("agrees with faculty"), not predictive ("makes better calls") — state which claim you're making. Circular-validation hazard: keep outcome data out of the faculty elicitation.
Statistical backbone. Frequentist BT MLE via choix (bootstrap CIs, Firth for separation) builds the reference scale. The examinee's calibration is the agreement between their implied worths and the reference (Spearman / concordance, or a Cooke-style calibration-×-information composite). Dynamic BT for learning curves. Bayesian-hierarchical BT is conditional (small examinee-N, partial pooling across a cohort), not default (ADR-CLIOH-07 draft; Playbook §7). Report the reference scale's test-retest reliability — a noisy answer key makes every calibration score meaningless.
For researchers — reporting, IRB & grant language
Reporting standards. Educational-measurement reporting (validity-evidence framing per Standards for Educational and Psychological Testing); AERA/APA/NCME validity argument; SQUIRE if framed as program QI. Map to EPA/milestone frameworks for translation (ten Cate 2005, Med Educ — EPAs use direct entrustment scoring, not pairwise BT; that gap is exactly what Calibration CLIOH fills). LitGuard every citation; DLRP Zotero mirror.
IRB / ethics. Higher-stakes than the internal-truth archetypes because it scores people with potential consequences for their training/careers. Human-subjects (examinee data); QI-vs-research determination up front. Fairness: a faculty-consensus norm can encode the panel's collective bias — interrogate it before any consequential use, and never make high-stakes pass/fail decisions off a normative scale lacking outcome-validity evidence. Examinee data governance and consent are real; no PHI in items; no unverified IRB claim in the UI. Be explicit with learners about what the score does and does not mean.
Grant-application language.
"Cooke's Classical Model of structured expert judgment scores experts against seed questions of known truth and weights them by a calibration-by-information product (Cooke, Experts in Uncertainty, Oxford University Press 1991; Colson & Cooke, Review of Environmental Economics and Policy 2018), an approach since applied to health (Hald, Aspinall, Cooke et al., WHO foodborne-disease attribution, PLOS ONE 2016). In parallel, Adaptive Comparative Judgement has become the dominant pairwise method for assessing student work in education, with documented reliability advantages (Verhavert, Bouwer, Donche & De Maeyer, Assessment in Education 2019), and calibrated judgment has been shown to be a learnable, trainable trait (Mellers et al., Psychological Science 2014). Calibration CLIOH unifies these: it builds a Bradley–Terry faculty-consensus reference scale over standardized pediatric-orthopaedic cases, then scores individual trainees by the agreement of their implied worths with that reference — yielding a continuous, calibrated assessment of surgical judgment that complements the entrustment-level EPA framework (ten Cate, Medical Education 2005) and tracks individual learning curves across residency via time-varying Bradley–Terry modeling."