Captured from the live deployment, not a mockup
01 · The problem
An LLM's self-reported confidence is not a probability, so it cannot decide what auto-publishes. Replaced it with a logistic calibrator over 11 evidence features fitted on held-out data, choosing the threshold to hit a measured precision floor.
02 · How it works
- 01
Traded auto-publish rate for correctness deliberately: publishing 80% instead of 100% moved precision on published values from 69.2% to 98.7%, safety-critical precision from 81.3% to 100%, and wrong values published from 28 to 1.
- 02
Kept the pipeline fully working with the LLM switched off, proving the model adds recall without being load-bearing for correctness.
- 03
Replaced an LLM's self-reported confidence with a logistic calibrator over 11 evidence features fitted on held-out data, picking the auto-publish threshold to meet a stated precision floor.
- 04
Publishing 80% rather than 100% moved precision on published values from 69.2% to 98.7% and cut wrong values published from 28 to 1, with safety-critical precision reaching 100%.
03 · What it cost, and what it returned
Against an extract-everything baseline on 98 hand-transcribed labels, publishing 80% instead of 100% lifted precision on published values from 69.2% to 98.7%, cut wrong published specs from 28 to 1 and calibration error from 0.308 to 0.037, with the one remaining error disclosed.