Naive Bayes Counts Twice

Question

Naive Bayes assumes features are independent given the class. Show with a tiny computation what happens when you feed it the same feature twice, generalize to correlated features, and explain why NB can still rank well while its probabilities are junk.

Accepted Answer

The mechanism. NB computes posterior odds as prior odds times a product of per-feature likelihood ratios: fracP(y=1mid x)P(y=0mid x) = fracpi_1pi_0prod_jfracP(x_j mid y=1)P(x_j mid y=0). The tiny computation. Take even prior odds and one feature with likelihood ratio 2 → posterior odds 2 (p = 2/3). Feed the identical feature in twice: odds 2 times 2 = 4 (p = 0.8). Ten copies: 2^10 = 1024 (p approx 0.999). Same single piece of evidence, near-certainty manufactured by bookkeeping. Correlated features are partial duplicates. Ten momentum variants that share 90% of their information get multiplied as if they were ten independent witnesses, systematic over-counting → posteriors saturating at 0/1. The probabilities are structurally overconfident whenever features share information, which in finance is always (signals come in correlated families by construction). Why ranking often survives. The over-counting is a roughly monotone distortion of the score: if it inflates both classes' evidence symmetrically, the order of scores across examples is preserved even as their values are nonsense, decent AUC, terrible calibration. Hence the classic fix: keep NB as a ranker and append a calibration

Naive Bayes Counts Twice

Problem

Your answer

Hints