Exploratory data analysis (EDA)

EDA is the first honest conversation you have with a dataset. The goal is not pretty charts; it is to find out what the data actually contains, what is broken in it, and which questions it can answer. Run the same checklist every time so nothing is missed under deadline pressure.

First contact

import pandas as pd

df = pd.read_csv("sales.csv", parse_dates=["date"])
df.info()                      # types & missing values
df.describe(include="all")     # summary statistics
df.isna().mean().sort_values(ascending=False)   # % missing
df["region"].value_counts()
df.groupby("region")["revenue"].sum().sort_values()
df.select_dtypes("number").corr()

Read info() closely. A revenue column typed as object means stray text such as "NA" or "1,20,000" is hiding in it. A date column still typed as object means parsing failed for some rows.

The EDA checklist

• Shape, types and missing values
• Duplicates and impossible values (negative ages, future dates)
• Distributions & outliers of key numeric columns
• Relationships with the target variable
• Time trends and seasonality

Check the things that break analyses

print(df.duplicated().sum())
print(df.duplicated(subset=["bill_id", "product"]).sum())
print(df[df["qty"] <= 0].shape)
print(df["city"].str.strip().str.title().value_counts().head(10))
print(df["date"].min(), df["date"].max())

Typical findings in Indian retail exports: "Bengaluru" and "Bangalore" as separate cities, returns entered as negative quantities, and a stray 1970 or 2099 date from a broken import.

Univariate, then bivariate

Column type	Look at	Typical chart
Numeric	Mean vs median, spread, skew	Histogram, box plot
Categorical	Number of levels, rare levels	Sorted bar chart
Date	Range, gaps, seasonality	Line chart by month
Numeric vs numeric	Correlation, shape	Scatter plot
Numeric vs category	Group means and spread	Box plot by group

When the mean is far above the median — order values usually are — the average is a poor summary. Report the median and a percentile range instead.

Outliers: investigate before deleting

q1, q3 = df["amount"].quantile([0.25, 0.75])
iqr = q3 - q1
suspect = df[(df["amount"] < q1 - 1.5 * iqr) | (df["amount"] > q3 + 1.5 * iqr)]
print(suspect[["bill_id", "city", "amount"]].head())

A ₹4 lakh bill may be a data entry error with an extra zero, or a genuine bulk wedding order. The first should be fixed, the second kept and possibly analysed separately.

Write down what you found

End every EDA with short notes: row count and date range, columns you cannot trust, decisions you took (rows dropped, values capped) and the questions the data cannot answer. This is what saves you when someone queries a number three months later.

Common mistakes

• Looking only at describe() and missing that a column is the wrong type.
• Dropping outliers automatically instead of investigating a handful of them.
• Reading correlation as causation; ice-cream sales and fan sales correlate through summer.
• Doing EDA on a small sample and assuming the full table behaves the same.