Accuracy Theater

Question

A model predicts whether a resting order gets filled within one second; fills happen 0.5% of the time. A colleague reports "99.5% accuracy." What's the first question to ask, which metrics actually mean something here, and how should the decision threshold be chosen?

Accepted Answer

First question: what does the trivial baseline score? Predicting "no fill" for every order achieves exactly 99.5% accuracy, the colleague's headline is indistinguishable from a model that learned nothing. Under heavy imbalance, accuracy ≈ majority-class prevalence; it measures the dataset, not the model. Metrics that carry information here: - Precision and recall on the rare class, at the threshold you'd act on, "of orders we flagged as likely fills, how many filled" and "of actual fills, how many we caught." - PR curve / average precision: base-rate-aware summary; far more honest than ROC under imbalance (ROC's false-positive rate divides by the enormous negative pool and flatters everyone). - Log-loss / calibration: if the output feeds an expected-value computation, probability quality is the real product. - Precision@k or lift: when you can only act on the top few percent anyway. Threshold from costs, not from 0.5. The optimal rule is expected-value: act when hat p cdot B > (1-hat p)cdot C, i.e. threshold = fracCB + C from the actual benefit B and cost C of the action. In this setting the asymmetry is concrete: predicting a fill that doesn't come costs queue priority or spread p

Accuracy Theater

Problem

Your answer

Hints