Why One Tree Is a Gamble

Question

Deep decision trees are described as low-bias, high-variance. Explain both halves precisely, in particular the mechanism that makes the variance so much worse than for other flexible models, and what the two standard remedies are.

Accepted Answer

Low bias. A deep tree partitions feature space into axis-aligned boxes with a constant per box; grown far enough it puts one sample per leaf and drives training error to zero. The model class can approximate any bounded function, there is essentially no systematic error left, only the other kind. High variance, three stacked mechanisms: 1. Greedy hierarchical instability (the big one). Splits are chosen sequentially by a noisy criterion. Perturb a handful of training rows and a different split can win at the root; everything below inherits a different sub-problem, so entire subtrees reorganize. Errors don't average across the structure, they cascade down it. This is qualitatively worse than, say, a polynomial fit, where perturbing data wiggles coefficients smoothly. 2. Few samples per leaf. Terminal predictions are means of a handful of points, high-variance estimates by construction, and exactly where financial label noise does maximal damage. 3. Hard thresholds. Predictions jump discontinuously when a feature crosses a split, so small input noise produces large output changes near boundaries. Remedies. - Regularize the single tree: depth limits, minimum leaf size, cost-complexity

Why One Tree Is a Gamble

Problem

Your answer

Hints