Train/test split & your first model

The whole point of a model is to work on rows it has never seen. So the first discipline you learn is to hide some data from yourself before training, and only look at it at the end.

Why hold out data

A model with enough capacity can memorise the training set and score 100% on it while being useless in production. Splitting the data gives you an honest estimate of performance on new customers.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

df = pd.read_csv("churn.csv")          # prepaid telecom subscribers
X = df.drop(columns="churned")
y = df["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

model = RandomForestClassifier(n_estimators=300, random_state=42)
model.fit(X_train, y_train)

pred = model.predict(X_test)
print(accuracy_score(y_test, pred))
print(classification_report(y_test, pred))

Three arguments do the real work:

Always beat a baseline

An accuracy of 0.86 means nothing until you know what "predict the majority class every time" scores.

from sklearn.dummy import DummyClassifier

base = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("baseline:", base.score(X_test, y_test))
print("model   :", model.score(X_test, y_test))

If the baseline is 0.85 and your model is 0.86, you have built an expensive coin flip.

Reading classification_report

• Precision for class 1: of the subscribers flagged as churning, what share actually churned.
• Recall: of the subscribers who actually churned, what share you caught.
• F1: their harmonic mean, useful when you need one number.
• Support: how many test rows each class had. Tiny support makes the other numbers unstable.

Train score vs test score

Common mistakes

• Leakage: leaving a column that encodes the answer, such as cancellation_reason or a post-churn flag. If your first model scores 0.99, hunt for leakage before celebrating.
• Splitting after scaling or imputing on the full dataset. Fit those on the training part only, ideally in a Pipeline.
• Shuffling time-ordered data. For forecasting, train on earlier months and test on later ones.
• Splitting rows when the unit is a customer with many rows — the same person ends up in both sets. Use GroupShuffleSplit.
• Tuning against the test set repeatedly. That turns it into a second training set; keep a separate validation split.