Hyperparameter tuning, cross-validation & imbalanced data

A model's parameters (weights, tree splits) are learned from data. Hyperparameters (tree depth, learning rate, C) are chosen by you. Choosing them on the test set is cheating; cross-validation gives you an honest way.

K-fold cross-validation

Split the training data into k folds; train on k-1 folds and validate on the remaining one, rotating so that every fold is used once for validation. The average score is far more stable than a single split.

from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring="average_precision")
print(scores.mean().round(3), scores.std().round(3))

Grid search vs random search

GridSearchCV tries every combination, which is fine for small grids. RandomizedSearchCV samples a fixed number of combinations from distributions and usually finds good settings much faster.

from scipy.stats import loguniform, randint
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import HistGradientBoostingClassifier

search = RandomizedSearchCV(
    HistGradientBoostingClassifier(random_state=42),
    param_distributions={
        "learning_rate": loguniform(0.01, 0.3),
        "max_leaf_nodes": randint(15, 64),
        "l2_regularization": loguniform(1e-4, 1.0),
    },
    n_iter=30, cv=cv, scoring="average_precision", n_jobs=-1, random_state=42,
)
search.fit(X_train, y_train)
print(search.best_params_, round(search.best_score_, 3))

Always put preprocessing inside a Pipeline so each fold scales/encodes using only its own training part. Tune pipeline steps with names like "model__max_depth".

Imbalanced data

When only 2% of UPI transactions are fraudulent, a model that predicts "not fraud" for everything is 98% accurate and completely useless. Tools that help:

1. Right metric: precision, recall, F1, PR-AUC (average_precision) instead of accuracy.
2. Class weights: class_weight="balanced" makes errors on the rare class cost more.
3. Resampling: undersample the majority or oversample the minority (for example SMOTE from the imbalanced-learn package), only inside training folds.
4. Threshold tuning: pick the probability cut-off that matches business costs.

import numpy as np
from sklearn.metrics import precision_recall_curve

proba = search.predict_proba(X_valid)[:, 1]
prec, rec, thr = precision_recall_curve(y_valid, proba)
ok = np.where(prec[:-1] >= 0.60)[0]              # need at least 60% precision
best = thr[ok[np.argmax(rec[ok])]]               # highest recall among those
print("threshold", round(best, 3))

Common mistakes

• Oversampling before splitting, which leaks copies of test rows into training.
• Reporting best_score_ as final performance. Evaluate once more on an untouched test set.
• Using shuffled KFold for time-ordered data.
• Tuning the threshold on the test set instead of a validation set.