Your challenge
1 · Inspect the signalsRead the customer's message, AI's predicted category, confidence, second-model disagreement, and possible impact.
2 · Pick up to tenA human review costs ₹1,000 per ticket. Your review budget is ₹10,000.
3 · Commit your IDsCopy your selected IDs. After everyone commits, the facilitator reveals the correct labels and costs.
Rules: Assume a paid human check catches and fixes a wrong category every time (a simplifying assumption). If an unchecked AI category is wrong, the stated fictional loss occurs. Correctly classified tickets have no error loss. Your aim is to minimise human review cost + unchecked error loss.
Data: Customer messages and model predictions come from a real BANKING77 benchmark. Impact amounts are fictional training assumptions, not real financial losses. Never enter real customer data.
Choose your reviews
Selected0 / 10
Review spending₹0
Budget remaining₹10,000
Showing 100 of 100 tickets
| Check? |
|---|
* Fictional impact, in Indian rupees. It is paid only if this ticket is wrong and you do not review it.
Public benchmark: sanand0.github.io/llmevals/confidence-calibration/ · Offline, standalone exercise