Archetype 10
Resource / Implementation Priority CLIOH
rank a portfolio of competing interventions, projects, or sites for finite funding or QI-deployment effort — turning a contentious budget meeting into a measured, defensible priority order.
- Evidence standard
- internal — the stakeholder consensus IS the answer (a strategic/values judgment, not an outcome prediction)
- What you compare
- Candidate interventions / projects / sites (competing for finite resources)
- Panel
- Mixed stakeholders + administration (not surgeon-experts alone)
- Output
- A prioritized portfolio (funded/deferred rank, or QI-deployment order) with worths + CIs
- Validation
- Test-retest; panel-composition sensitivity (essential — stakeholders have interests); convergence
What it's for
A CLIOH study in which a mixed stakeholder panel makes pairwise judgments over candidate investments (studies, interventions, QI initiatives, sites), and a Bradley–Terry model produces an interval-scale priority order to guide allocation of finite funding or implementation effort.
"Of the things competing for our limited dollars/bandwidth, which should we fund or deploy first — and can we defend that order to the people who didn't get funded?"
When to use it
(a) Finite resources force explicit prioritization across a portfolio; (b) the decision is strategic/values-laden and benefits from structured, transparent aggregation; (c) multiple stakeholders with different vantage points must be reconciled; (d) you want an auditable, reproducible rank rather than the loudest-voice-wins outcome of a meeting; (e) the items are comparable enough to be paired meaningfully.
When not to
(a) You're ranking scientific variables as a research finding, not investments for a decision → Discovery §1 (different purpose, expert-only panel); (b) you're ranking trial endpoints → Outcome-Importance §6 (queued); (c) a clean cost-effectiveness or ROI metric already settles the question (use it); (d) the "portfolio" is really one decision with a dominant criterion (just optimize that criterion); (e) the stakes and politics make any expert-elicited rank unusable — sometimes the honest answer is that this needs governance, not measurement.
What you get
A prioritized portfolio: ranked investments with worths and CIs, the per-criterion sub-rankings, the composite weighting made explicit, and the inter-stakeholder disagreement surfaced. Delivered as a decision aid for the funding/QI body. Aggregate-only display; no unverified claims.
A real example
- CORTICES annual research-portfolio prioritization (candidate first instance): rank the consortium's candidate studies/initiatives for the year against finite funding and bandwidth, using a mixed panel (surgeons, administrators, methodologists) and multi-criteria sub-BT (scientific impact × feasibility × strategic fit). The reusable artifact is the annual prioritization protocol.
- QI-deployment ordering: sequence a set of quality initiatives across sites by stakeholder-judged priority when they can't all launch at once.
- This is the archetype most likely to earn institutional/administrative buy-in, because its output is a budget decision aid.
Validation
Test-retest of the elicitation; panel-composition sensitivity (drop-one-out, and drop-one-stakeholder-group) — the single most important check, because a rank that flips when one interest group leaves is a political artifact, not a consensus; convergence with independent review. Pre-register the criterion weights before eliciting, so the composite isn't reverse-engineered to a desired answer.
Common pitfalls
(a) Panel composition driving the answer — with interested stakeholders this is the dominant failure mode; report drop-one-group sensitivity or the rank isn't trustworthy. (b) Collapsing multi-criteria into one judgment so impact silently swamps feasibility — use sub-BT. (c) Reverse-engineering the weights to a desired outcome — pre-register them. (d) Conflict-of-interest opacity — declare interests. (e) Strength-Spread failure — no anchors → compressed, uninformative priority scale. (f) Treating a values judgment as predictive — it's normative; don't claim the top-ranked project will have the best outcome. (g) Confusing it with Discovery §1 — same machinery, but here the items are investments, the panel is mixed, and the purpose is allocation.
How it works — show me the method
Item selection. Candidate interventions/projects/sites, scoped to be genuinely comparable. Strength-Spread §1.1 applies (no natural endpoints) — deliberately include a clearly-top-priority and a clearly-low-priority anchor so the rank discriminates and isn't compressed; anchors pre-tested at Stage 0 Gate A. Keep the slate to a tractable size; if the portfolio is huge, pre-screen with CLIOH-FIT before Stage 0.
What you compare. Decide up front whether the elicited worth is overall priority or a single criterion (impact, feasibility, cost-effectiveness, strategic alignment). Portfolio decisions are usually multi-criteria — see §7 for the sub-BT structure that keeps impact from silently swamping feasibility.
Comparison structure. Bare-vote, binary forced-choice pairwise, NO ties (ADR-CLIOH-03). BIBD pair allocation; ATRD Round 2 (TRE) on contested pairs (ADR-CLIOH-04). Resource-specific add-on — multi-criteria sub-BT (DR §10 add-on, adopted): run a separate pairwise pass per criterion (e.g., impact, feasibility, cost, alignment), fit a sub-BT per criterion, then combine the criterion worths into a composite priority with explicit, pre-registered weights — rather than asking raters to collapse everything into one "overall" judgment. This makes the trade-offs visible and auditable. Single-step back button (ADR-CLIOH-02).
Panel. Mixed stakeholders + administration — the distinguishing feature of this archetype. Include those who hold the resources, those who implement, and those affected, not surgeon-experts alone. Because stakeholders have interests, panel-composition sensitivity is not optional here (see §12). 15–30 raters; if N<15, Cooke-weighted aggregation.
External ground truth. Internal. The stakeholder consensus is the answer — this is a strategic/values judgment, not an outcome prediction, so there is no external gold standard to validate against. Validate by test-retest, panel-composition sensitivity, and convergence with independent strategic review. State that the claim is normative, not predictive.
Statistical backbone. Frequentist BT MLE via choix (bootstrap CIs, Firth) per criterion and/or overall. Composite priority = pre-registered weighted combination of criterion sub-BT worths. Bayesian-hierarchical BT is conditional (stakeholder subgroups, small N — and useful here for partial pooling across stakeholder groups with different vantage points), not default (ADR-CLIOH-07 draft; Playbook §7). Report disagreement between stakeholder groups as a finding, not noise.
For researchers — reporting, IRB & grant language
Reporting standards. Priority-setting / decision-analysis reporting. Position against the methodological cousins: CHNRI research-priority-setting (Rudan, J Global Health 2016) uses crowdsourced scoring of research questions; MCDA (endorsed by IQWiG for reimbursement decisions) and the Analytic Hierarchy Process (Saaty 1977; widely used in clinical-guideline development per Schmidt et al. 2015) use multi-criteria weighting; Best–Worst Scaling Case 1 / MaxDiff is the directly analogous prioritization method (Schuster et al., J Choice Modelling 2024). CLIOH's bare-vote pairwise + multi-criteria sub-BT is a clean, low-burden alternative. SQUIRE if framed as QI. LitGuard + DLRP.
IRB / ethics. Typically not human-subjects (an institutional/strategic decision among stakeholders) — but make the QI-vs-research determination up front if you intend to publish the method. No PHI; aggregate-only. The ethics center on transparency and conflict-of-interest: stakeholders rank things they may benefit from, so declare interests and report panel-composition sensitivity. No unverified IRB claim in the UI.
Grant-application language.
"Resource-allocation and research-priority-setting have well-established structured methods — CHNRI for global-health research priorities (Rudan, Journal of Global Health 2016), multi-criteria decision analysis endorsed by Germany's IQWiG for reimbursement, the Analytic Hierarchy Process in clinical-guideline development (Saaty 1977; Schmidt et al. 2015), and Best–Worst Scaling for healthcare prioritization (Schuster et al., Journal of Choice Modelling 2024). Resource/Implementation CLIOH applies the Bradley–Terry framework to this problem with a mixed-stakeholder panel and a multi-criteria sub-model: separate pairwise elicitations on impact, feasibility, cost, and strategic alignment are each fit with Bradley–Terry and combined into a transparent, pre-weighted composite priority — yielding a reproducible, auditable portfolio rank that surfaces, rather than hides, the trade-offs and the disagreements between stakeholder groups."