Archetype 2
Educational / Training CLIOH
make the tacit priority order of senior faculty explicit and teachable — convert "you'll just know it when you've seen enough" into a measured, curriculum-ready hierarchy.
- Evidence standard
- internal — the faculty consensus IS the standard (normative/descriptive, never validated against patient outcomes)
- What you compare
- Standardized teaching scenarios / conditions / cases
- Panel
- Senior faculty (the people whose judgment is the teaching standard)
- Output
- A trainee curriculum + assessment (a teachable priority order + exam items)
- Validation
- Test-retest reliability; faculty-panel-composition sensitivity; convergence with later teaching/empirical work
What it's for
A CLIOH study that elicits senior-faculty pairwise judgments over standardized teaching scenarios and fits a Bradley–Terry model to produce an explicit, interval-scale priority order packaged as a teaching tool and assessment — making expert tacit knowledge transmissible.
"What is the faculty's collective priority order on this topic, expressed clearly enough that we can teach it and examine it — rather than hoping residents absorb it by osmosis?"
When to use it
(a) Senior faculty hold strong tacit judgment that is hard to articulate explicitly; (b) you want a curriculum or exam-item bank grounded in measured faculty consensus; (c) trainee judgment on the topic is inconsistent and you want a normative target to teach toward; (d) the topic is a matter of clinical priority/judgment, not pure factual recall; (e) you intend to pair it with a trainee assessment (→ H1).
When not to
(a) The deliverable is a research finding about which variables matter, not a teaching tool → Discovery §1 (same machinery, different purpose and packaging); (b) you want to score an individual against the standard → Calibration §5 (the natural partner — build the norm here, score there); (c) a validated factual exam already covers the competency; (d) you are tempted to validate the result against patient outcomes — don't: Educational's truth source is faculty consensus, and trying to outcome-validate it is the classic archetype-mismatch error (see §9); (e) the construct has a clean external gold standard the curriculum should teach to instead.
What you get
A teaching package: the measured faculty priority order, anchored worked exemplars, the ATRD rationales as teaching points, and an exam-item bank of standardized scenario pairs ready to deploy against trainees (→ §5). Aggregate-only display; no PHI in scenarios; no unverified claims in the instrument.
A real example
- CORTICES triage-priority teaching curriculum (candidate first instance): the attending faculty priority order from OR/ER-Priority, repackaged as a teaching hierarchy + exemplar cases for residents (the DR labeled this "CLIOH-ED Triage" as an education example). The reusable artifact is the curriculum + item bank.
- H1 Calibration × Educational (the high-yield combination): build the faculty norm here (§2), then deploy a calibration quiz against trainees (§5). One elicitation → both a teaching tool and an assessment. Target institutionalization for the AAOS OITE.
- ATRD rationales as teaching content: the targeted-rationale text surfaced during Round 2 is itself a curriculum asset — it captures the reasoning behind the priority order, which is what trainees most need.
Validation
Test-retest of the faculty elicitation; panel-composition sensitivity; convergence with independent teaching consensus or guidelines; and — when paired with §5 — evidence that trainees move toward the faculty norm after instruction (a learning signal, not an outcome claim). Pre-register the reliability analysis.
Common pitfalls
(a) Outcome-validating an internal-truth instrument — the single most common archetype-mismatch error; Educational's standard is the faculty, not patient outcomes. (b) Using trainees as the panel — they're the audience, not the source. (c) A noisy faculty scale — report test-retest before building any curriculum or exam on it. (d) Strength-Spread failure — scenarios clustered at the easy end produce an undiscriminating curriculum and exam. (e) Confusing it with Discovery §1 — same machinery, but the deliverable here is a teaching tool, packaged and validated as such. (f) Overclaiming — a measured faculty norm is normative, not a guarantee of better outcomes.
How it works — show me the method
Item selection. Standardized teaching scenarios / conditions / cases chosen for pedagogical coverage of the topic. Strength-Spread §1.1 applies — the scenario set should span clearly-high, borderline, and clearly-low priority exemplars so the taught hierarchy discriminates across the whole range and so exam items aren't all clustered at the easy ends. Anchor scenarios pre-tested at Stage 0 Gate A.
What you compare. The construct is the teachable judgment dimension (operative priority, urgency, management aggressiveness, "what would a senior attending do first"). Keep scenarios realistic and self-contained so a trainee can later face the same items in the assessment half (H1) without ambiguity.
Comparison structure. Bare-vote, binary forced-choice pairwise, NO ties (ADR-CLIOH-03). BIBD pair allocation; ATRD Round 2 (TRE) on contested pairs (ADR-CLIOH-04) — the rationales surfaced in ATRD are themselves teaching gold (they make the why explicit, which is the whole point of an educational instrument). Single-step back button (ADR-CLIOH-02). (Adaptive Comparative Judgement / entropy-driven pair selection is a future enhancement, not in the locked core.)
Panel. Senior faculty — the clinicians whose judgment defines the teaching standard; 15–30 (target 25–30; if N<15, Cooke-weighted aggregation). This is deliberately not a trainee panel: trainees are the audience in §2 and the examinees in §5, never the source of the norm.
External ground truth. Internal. The faculty consensus is the standard; there is no external gold standard to validate against, because the goal is to transmit expert judgment, not to predict outcomes. Validate by test-retest reliability, faculty-panel-composition sensitivity (drop-one-out), and convergence with later teaching/empirical work. State plainly that the claim is normative/descriptive, not predictive. Attempting outcome validation here is a category error (DR §archetype-mismatch warning).
Statistical backbone. Frequentist BT MLE via choix (bootstrap CIs, Firth for separation) yields the faculty priority order = the teachable scale. Report test-retest reliability prominently (a noisy scale makes a poor curriculum and an indefensible exam). Bayesian-hierarchical BT is conditional (subgroup curricula, small faculty N), not default (ADR-CLIOH-07 draft; Playbook §7). The scale and its anchored exemplars become the curriculum content and the item bank.
For researchers — reporting, IRB & grant language
Reporting standards. Medical-education reporting (validity-evidence framing per Standards for Educational and Psychological Testing); map outputs to competency frameworks for translation — note that Entrustable Professional Activities use direct entrustment scoring, not pairwise comparison (ten Cate, Medical Education 2005), which is precisely the methodological gap a BT-based Educational instrument fills. SQUIRE if framed as curricular QI. LitGuard every citation; DLRP Zotero mirror.
IRB / ethics. The faculty elicitation is expert-opinion (often QI/exempt), but make the QI-vs-research determination up front, especially if trainee data will be collected in the §5 half. No PHI in teaching scenarios; aggregate-only display. Be transparent with trainees about what the norm represents (measured faculty consensus, not ground truth). No unverified IRB claim in the UI.
Grant-application language.
"Comparative Judgement — introduced to educational assessment by Pollitt (2004, 'Let's Stop Marking Exams') and shown in meta-analysis to yield highly reliable rankings of student work (Verhavert, Bouwer, Donche & De Maeyer, Assessment in Education 2019) — is now standard in national writing assessment (Wheadon et al. 2020). Educational CLIOH brings this validated pairwise paradigm to surgical training: it elicits senior-faculty pairwise judgments over standardized pediatric-orthopaedic scenarios and fits a Bradley–Terry model to make the faculty's tacit priority order explicit, interval-scaled, and teachable. Unlike the Entrustable Professional Activities framework, which relies on direct entrustment scoring (ten Cate, Medical Education 2005), the Bradley–Terry approach yields a continuous, reproducible curriculum and assessment scale. Paired with Calibration CLIOH, a single faculty elicitation produces both a teaching tool and a trainee assessment (Hybrid H1) — the highest-yield application for residency programs and a candidate for institutional adoption (e.g., the AAOS Orthopaedic In-Training Examination)."