Hybrid H1
Calibration × Educational CLIOH
from one faculty elicitation, produce both a teaching tool (Educational §2) and a trainee assessment (Calibration §5).
- Evidence standard
- internal (both deliverables) — faculty consensus is the standard for the curriculum AND the answer key for the trainee score
- Panel
- Faculty build the reference; trainees are taught (§2) and examined (§5) — never the source of the norm
- Validation
- Test-retest of the faculty scale; panel-composition sensitivity; learning signal (trainees move toward the norm after instruction)
What it's for
A single faculty pairwise elicitation, fit with Bradley–Terry, yields an interval-scale priority order that serves simultaneously as (a) a curriculum + exam-item bank (the Educational deliverable) and (b) the reference "answer key" against which individual trainees are scored (the Calibration deliverable).
The two parents share the exact same artifact — the faculty worth scale over standardized scenarios. Educational packages it as content to teach; Calibration uses it as the target to score against. Running them separately would elicit the same faculty judgments twice. H1 collapses that into one Stage-0-gated elicitation with two downstream products, which is why it is the most resource-efficient and highest-adoption application for a training program.
"What does the faculty collectively prioritize (so we can teach it), and how close is each trainee to that standard (so we can assess and track them)?"
When to use it
Use when a program wants both a defensible curriculum and a trainee assessment on the same judgment dimension, and the faculty norm is stable enough to anchor both. Don't when you only need one half (use the parent), when the construct is factual recall rather than judgment, or when the stakes demand a validated high-stakes certification instrument that a normative scale can't yet support (see §10/§12 of Calibration §5).
What you get
- Educational: the measured faculty priority order, anchored exemplars, ATRD rationales as teaching points, and a standardized exam-item bank.
- Calibration: per-trainee report (agreement with the norm + CI, where they diverge), and — for tracked cohorts — a learning curve climbing toward the faculty standard.
A real example
CORTICES faculty priority order → resident curriculum + calibration quiz. The attending priority order from OR/ER-Priority (already a Discovery §1 instrument) is repackaged as a teaching hierarchy (§2) and simultaneously used as the answer key for a resident calibration quiz (§5). Reusable artifact: a calibrated, anchored scenario bank serving both. Institutional target: AAOS OITE.
Validation
Test-retest + drop-one-out panel-composition sensitivity on the faculty reference; evidence that trainees move toward the norm after instruction (a learning signal); pre-register the scoring rule before any trainee data. If an LLM is ever used as a virtual examinee and κ<0.5 on calibration items, don't deploy it for that domain (DR threshold).
Common pitfalls
(a) Outcome-validating an internal-truth hybrid — the standard is the faculty, not patient outcomes. (b) Using trainees as the panel — they're audience/examinee, not source. (c) A noisy faculty scale — validate test-retest before building either deliverable on it. (d) Trainee data leaking into the norm. (e) High-stakes use without validity evidence (fairness — a faculty norm can encode collective bias). (f) Building the two halves as separate studies — the whole point is one elicitation.
How it works — show me the method
Comparison structure. Bare-vote, binary forced-choice pairwise, NO ties (ADR-CLIOH-03) over standardized scenarios; BIBD allocation; ATRD Round 2 (TRE) on contested pairs (ADR-CLIOH-04) — and the ATRD rationales are reused directly as teaching content. Single-step back button (ADR-CLIOH-02). The faculty pass builds the reference; trainees later complete the same standardized pair set, and their implied worths are compared to the reference for the calibration score. Faculty and trainee passes are kept strictly separate.
Panel. Faculty (reference builders): 15–30 senior experts (target 25–30; if N<15, Cooke-weighted aggregation). Trainees (audience + examinees): the learners, scored individually. Trainee data never feeds back into the norm.
Ground truth. Internal for both deliverables. The faculty consensus is the standard — for the curriculum and for the answer key. Do not outcome-validate (the classic archetype-mismatch error from Educational §2 §9). The optional distal outcome layer (from Calibration §9) only enters if the program wants to claim the calibration score is predictive, not merely normative — say which claim you're making.
Statistical backbone. Frequentist BT MLE via choix (bootstrap CIs, Firth) builds the faculty reference; report its test-retest reliability prominently (a noisy answer key poisons both deliverables). Trainee calibration = agreement of implied worths with the reference (Spearman/concordance, or a Cooke-style calibration-×-information composite). Dynamic / time-varying BT (Cattelan; Varin & Firth 2013) produces individual learning curves across residency. Bayesian-hierarchical BT is conditional (partial pooling across a trainee cohort).