The Dummy Variable Trap

Question

You one-hot encode a categorical feature with 12 levels (say, the month of the year) for a linear regression that includes an intercept. How many indicator columns should you create, and what exactly goes wrong if you create one per level?

Accepted Answer

Create mathbf11 columns, one fewer than the number of levels. The mechanism. With all 12 indicators, every row's indicators sum to exactly 1, because each observation belongs to exactly one month. So the sum of the 12 columns equals the intercept's all-ones column: a perfect linear dependence. X^top X is then singular, OLS has no unique solution, and any software that "succeeds" is silently picking one of infinitely many coefficient vectors (or dropping a column for you). Interpretation after the fix. Dropping one level makes it the baseline: each remaining coefficient is that month's mean shift relative to the baseline month, and the intercept absorbs the baseline's level. The fitted values are identical for any choice of dropped level; only the parameterization changes, which is why coefficient values across two such models can look wildly different while predicting identically. Where it actually bites. Ridge or lasso regularization technically rescues the full 12-column encoding (the penalty breaks the tie), and tree models never care. But in unregularized regressions, including the quick OLS diagnostics quants run constantly, the trap produces exploding standard errors or solve

The Dummy Variable Trap

Problem

Your answer

Hints

More Machine Learning to train on