Decision trees, random forests & gradient boosting

For tabular business data (sales, customers, transactions) tree-based models are usually the strongest out-of-the-box choice. They need no scaling, handle non-linear patterns and capture feature interactions on their own.

Decision trees

A decision tree learns a sequence of if-else questions: Is monthly spend > INR 2,000? Is tenure < 6 months? At each node it picks the split that makes the child groups as "pure" as possible, measured by Gini impurity or entropy for classification and squared error for regression.

Trees are easy to read, but a fully grown tree memorises the training data. Control it with max_depth, min_samples_leaf and ccp_alpha (cost-complexity pruning).

from sklearn.tree import DecisionTreeClassifier, export_text

tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=20, random_state=42)
tree.fit(X_train, y_train)
print(export_text(tree, feature_names=list(X_train.columns)))

Random forests: many trees, averaged

A random forest trains hundreds of trees, each on a bootstrap sample (rows drawn with replacement) and considering only a random subset of features at each split. Averaging many decorrelated, overfit trees cancels out much of their variance. This idea is called bagging.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=300, max_features="sqrt",
                            min_samples_leaf=5, n_jobs=-1, random_state=42)
rf.fit(X_train, y_train)
top = sorted(zip(rf.feature_importances_, X_train.columns), reverse=True)[:5]
print(top)

Gradient boosting: trees that fix each other's mistakes

Boosting builds trees sequentially. Each new shallow tree is fitted to the errors (gradients of the loss) of the ensemble so far, and its contribution is scaled by a learning_rate. This mainly reduces bias, and it is why XGBoost, LightGBM and CatBoost dominate tabular competitions. scikit-learn ships a fast histogram-based version:

from sklearn.ensemble import HistGradientBoostingClassifier

gb = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=500, max_leaf_nodes=31,
    early_stopping=True, validation_fraction=0.1, random_state=42,
)
gb.fit(X_train, y_train)          # handles NaN natively
print(gb.n_iter_, gb.score(X_test, y_test))

Choosing between them

A practical path: start with a random forest as a robust baseline, then try gradient boosting with early stopping and tune the learning rate and number of leaves.

Common mistakes

• Trusting impurity-based feature_importances_ blindly. They favour high-cardinality features, so cross-check with sklearn.inspection.permutation_importance on validation data.
• Setting a high learning rate with many iterations; boosting then overfits quickly.
• One-hot encoding a column with thousands of categories (like PIN codes) for trees. Group rare values or use target / ordinal encoding.
• Scaling features for trees. It is harmless but pointless, because splits depend only on the order of values.