Evaluating models

"Is the model good?" is never answerable on its own. Good for what decision, and compared to what alternative? Metrics turn that question into a number you can defend in a review meeting.

The confusion matrix underneath everything

For classification, every prediction falls into one of four boxes:

	Predicted no	Predicted yes
Actually no	True negative	False positive
Actually yes	False negative	True positive

from sklearn.metrics import confusion_matrix, classification_report

pred = clf.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))

• Precision = TP / (TP + FP): when we say yes, how often are we right?
• Recall = TP / (TP + FN): of all real yes cases, how many did we catch?
• F1 = harmonic mean of the two.

Which one matters is a business decision. For a retention campaign that costs ₹60 per call, high recall is worth some wasted calls. For blocking a customer's card, a false positive is expensive, so precision rules.

Metrics by task

Task	Metrics	Read it as
Classification	Accuracy, Precision, Recall, F1	Share of decisions that were right
Ranked classification	ROC-AUC, PR-AUC (average_precision)	Quality of the score, at every threshold
Regression	MAE, RMSE, R-squared	Typical error, in the target's own units

from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

print(mean_absolute_error(y_test, pred))       # avg error in INR
print(root_mean_squared_error(y_test, pred))   # punishes big misses
print(r2_score(y_test, pred))                  # share of variance explained

MAE of ₹4,200 on an average bill of ₹48,000 is roughly 9% off — that sentence is what the business understands. RMSE is always at least MAE; a large gap means a few very bad predictions.

Threshold and imbalance

predict uses a 0.5 cut-off you never agreed to. Score with probabilities and choose the threshold yourself.

from sklearn.metrics import roc_auc_score, average_precision_score

proba = clf.predict_proba(X_test)[:, 1]
print(roc_auc_score(y_test, proba), average_precision_score(y_test, proba))
flag = proba > 0.30

On imbalanced data such as 2% fraud, accuracy is meaningless (predict "no" always and score 98%). ROC-AUC still looks flattering because true negatives dominate; PR-AUC is the honest summary.

One split is not enough

from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X, y, cv=5, scoring="f1")
print(scores.mean().round(3), scores.std().round(3))

Report the mean and the standard deviation. If two models differ by less than one standard deviation, you cannot claim one is better.

Common mistakes

• Reporting accuracy on imbalanced data.
• Comparing metrics computed on different splits or different row filters.
• Optimising a metric that does not match the cost of errors.
• Forgetting a business baseline: last year's rule, or the current manual process.
• Quoting R-squared without checking residuals for a pattern the model missed.