The Trust Inversion: When Your Most Confident AI Predictions Are the Most Wrong

Most selective-prediction systems rest on one assumption: higher confidence means more likely correct, so you accept the high-confidence answers and abstain on the rest. For much of the input space that holds. In the hardest cases, we measured it breaking — and breaking in a specific, exploitable way.

The measurement

REMORA classifies each decision into a phase based on the structure of model disagreement. In the 'critical' phase — where consensus sits near a boundary — the trust score anti-correlated with correctness on a set of real-oracle items:

• Low-trust critical items (trust < 0.10): 71.4% correct (N = 21).
• High-trust critical items (trust ≥ 0.10): 27.3% correct (N = 11).

Trusting the confident answers in this regime would have been worse than a coin flip. And a standard conformal calibration aimed at a 5% risk target collapsed here — 100% observed risk, coverage falling to zero — because the assumption that calibration and test data are exchangeable is violated across the phase boundary.

Why this happens

In the critical phase, strong agreement is a sign of correlated error — a shared prior, a shared training blind spot, a plausible-but-wrong framing that all models latch onto — rather than of independent convergence on truth. The confidence is real; what it is confident about is the consensus, not the answer. A safety system that reads that confidence at face value is most certain exactly when it should be most cautious.

Route around it, don't trust it

The wrong response is to try to 'fix' the model into being well-calibrated everywhere. The right response is to treat the inversion as a selection-criterion reversal. Where the data say confidence is anti-informative, REMORA inverts the score for critical-phase items, calibrates the threshold on the inverted score, and hard-rejects anything above the groupthink boundary regardless of the calibration result.

Combining the ordered-phase items with the inverted low-trust critical items extends coverage to 22.1% at 85.0% accuracy (Wilson CI [77.5%, 90.3%]) — a coverage gain bought without lowering the safety floor, in precisely the regime where naive trust-based routing produces zero usable coverage.

The general lesson

A safety system's job is not to be confident. It is to know when its confidence is worthless — and route around it.

Calibration is not a global property you can assume. It is a regime-dependent one you have to measure. When you find a regime where confidence betrays you, the honest engineering move is to document the betrayal and design a selection rule that survives it — not to paper over it with a number that looks reassuring on a dashboard.