Hybrid H3

Classification × AI-Calibration CLIOH

turn an expert classification worth scale (Classification §3) into the continuous regression target that trains or benchmarks a clinical AI (AI-Calibration §9).

HybridCandidatei-ROP (the templatePMID 35157950); CORTICES fracture-grading case bank → grading scale + AI training target (candidate)

Evidence standard: external (both deliverables) — validate against outcomes/labels; AI side is two-layer (worth scale proximal, outcomes distal)
Panel: Domain experts (the grading standard) + the ML/data-science team
Output: An interval-scale expert grading scale over real cases
Validation: ROC/AUC vs outcomes; model–worth concordance; case-mix audit; SaMD gating before deployment

What it's for

A single expert case-grading elicitation, fit with Bradley–Terry, produces an interval-scale worth over real cases that functions simultaneously as (a) a classification/grading scale and (b) the continuous regression target against which a clinical AI is trained or benchmarked.

A categorical label is a coarse, lossy training signal; a BT worth scale is continuous and interval-scaled, which is a far richer target for a model and a far cleaner benchmark. The same expert effort that yields a grading scale yields the AI's target — one case bank, two deliverables. This is the methodological core of why pairwise-derived scales beat ordinal labels for AI (Pedersen knee-OA, Phelps image-quality, i-ROP all show the gain).

"What is the expert grading scale for these cases, and does our model reproduce it — measured on the experts' own continuous ruler rather than a coarse label?"

When to use it

Use when you want both an expert grading scale and a model calibrated/trained to it, on the same real-case bank, with outcome linkage available. Don't when a cheap hard outcome label already exists (train on it directly), when no model is involved (use Classification §3), or when you're ranking variables rather than grading cases (Discovery §1).

What you get

Classification: an interval-scale expert grading scale over real cases (with cutpoints if a categorical grade is needed).
AI-Calibration: the continuous regression target + a model calibrated/trained to it, with model–worth concordance and outcome-layer AUC reported.

A real example

i-ROP (the template): 34 ICROP3 experts, pairwise/Elo comparisons → a vascular severity scale; a deep-learning model validated against it at correlation 0.90 with the expert plus-disease classification (Ophthalmology 2022, PMID 35157950 — not "Bellsmith"). CORTICES analogue (candidate): a fracture-grading case bank yielding both an expert grading scale and the training target for a grading model. Reusable artifact: a labeled, expert-graded case bank that is simultaneously a clinical scale and an AI dataset.

Validation

Outcome-linked ROC/AUC for the grading scale; model–worth concordance + distal-outcome AUC for the model; case-mix/spectrum audit (the Classification + AI families' shared threat); drift monitoring for production models; pre-register the benchmark. No deployment without separate IRB + prospective validation + SaMD-grade QMS.

Common pitfalls

(a) Circular validation — experts must not see model output/outcomes while rating. (b) Letting the proximal worth scale masquerade as ground truth — always report the distal outcome layer. (c) Incommensurable targets — define the model↔worth mapping before elicitation. (d) Case-mix bias — a bank missing the borderline cases yields a scale and a model that look good where it doesn't matter. (e) The model inheriting the experts' blind spots — the scale is only as good as the panel; report panel-composition sensitivity. (f) Over-claiming SaMD status before separate validation.

How it works — show me the method

Comparison structure. Bare-vote, binary forced-choice pairwise "which case is more [construct]?", NO ties (ADR-CLIOH-03); BIBD allocation; ATRD Round 2 (ADR-CLIOH-04); single-step back button (ADR-CLIOH-02). The resulting worths are the classification scale and the regression target. Define the model↔worth mapping before elicitation so the target is commensurable with what the model emits.

Panel. Domain experts whose grading is the standard (15–30) plus the ML/data-science team who own the model and the target definition — included early to prevent an elegant scale that's incommensurable with the model output.

Ground truth. External for both. The Classification scale validates against outcomes/labels; the AI side is two-layer (the expert worth scale is the proximal standard the model is scored against; an outcome dataset is the distal standard). Report agreement at both layers; never let the proximal scale pose as ground truth. Circular validation (experts seeing model output or outcomes while rating) is the cardinal sin.