Pipelines & preprocessing

Real tables are messy: numbers on wildly different scales, text categories, missing values. Models need numbers. A Pipeline bundles every cleaning step with the model into a single object that you fit once and reuse everywhere.

The leakage problem pipelines solve

If you scale the whole dataset before splitting, the mean and standard deviation used for scaling already contain information from the test rows. Your score looks better than reality. A pipeline fixes this automatically: inside each fit, the transformers learn from the training part only.

ColumnTransformer: different treatment per column

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression

num = ["age", "monthly_bill"]
cat = ["plan", "city"]

num_steps = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
cat_steps = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore", min_frequency=20)),
])

pre = ColumnTransformer([("num", num_steps, num), ("cat", cat_steps, cat)])
clf = Pipeline([("pre", pre), ("model", LogisticRegression(max_iter=1000))])
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))

handle_unknown="ignore" is not optional in production: a new city will appear one day, and without it predict raises an error. min_frequency=20 folds rare categories into one "infrequent" column, which keeps a PIN-code style feature from exploding into thousands of columns.

Choosing transformers

Select columns by type instead of listing them by hand when the schema keeps changing:

from sklearn.compose import make_column_selector as selector

pre = ColumnTransformer([
    ("num", num_steps, selector(dtype_include="number")),
    ("cat", cat_steps, selector(dtype_include=object)),
])

Inspecting and tuning a pipeline

names = clf.named_steps["pre"].get_feature_names_out()
print(len(names), names[:5])

from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(clf, {"model__C": [0.1, 1, 10]}, cv=5, scoring="f1")
grid.fit(X_train, y_train)
print(grid.best_params_)

Step names plus a double underscore address any nested parameter, so the whole pipeline is tuned as one unit and each cross-validation fold preprocesses itself correctly.

Common mistakes

• Calling fit_transform on the test set. Use transform — or let the pipeline handle it.
• Dropping rows with missing values instead of imputing, then meeting a missing value in production anyway.
• One-hot encoding a high-cardinality column without min_frequency, which creates thousands of sparse columns.
• Scaling the target variable and forgetting to invert it before reporting rupees.
• Saving the model but not the preprocessing, so the serving code re-implements it slightly differently.