Descriptive statistics

Before any model or test, you describe what you have. Descriptive statistics compress thousands of rows into a few numbers a manager can hold in their head - and choosing the wrong few numbers is how dashboards mislead.

Measures of centre

import statistics as st

bills = [12, 15, 15, 18, 20, 95]      # ₹ hundred, one sweets-shop counter
print(st.mean(bills), st.median(bills), st.mode(bills))   # 29.2  16.5  15

The mean of ₹2,920 describes nobody: five of the six bills are below ₹2,000. One bulk festival order dragged it up. Salaries, property prices, claim amounts and order values are all right-skewed, so report the median, and show the mean beside it only when you also show the skew.

Measures of spread

Two shops can both average ₹4 lakh a month, yet one is steady and the other swings between ₹1 lakh and ₹9 lakh. Spread tells them apart.

• Range - max minus min. One outlier ruins it.
• Variance - average squared distance from the mean, in squared units.
• Standard deviation - the square root of variance, back in rupees. Roughly the typical distance from the mean.
• IQR - Q3 minus Q1, the width of the middle 50%. Immune to outliers.
• Coefficient of variation - std / mean, a unit-free way to compare volatility between a ₹5,000 item and a ₹5 lakh item.

Note the n-1 divisor: statistics.stdev and df.std() compute the sample standard deviation (an unbiased estimate of the population value), while statistics.pstdev and numpy.std default to the population formula. On 500+ rows the difference is negligible; on 8 rows it is not.

Shape, percentiles and outliers

import pandas as pd

df = pd.read_csv("invoices.csv")           # column: amount_inr, city
print(df["amount_inr"].describe())
print(df["amount_inr"].skew())             # > 0 means a long right tail

q1, q3 = df["amount_inr"].quantile([0.25, 0.75])
iqr = q3 - q1
outliers = df[(df.amount_inr < q1 - 1.5 * iqr) | (df.amount_inr > q3 + 1.5 * iqr)]
print(len(outliers), "unusual invoices")

print(df.groupby("city")["amount_inr"].agg(["count", "median", "std"]))

Percentiles often beat averages in operations: an SLA of "95% of deliveries within 48 hours" (the 95th percentile) is a promise; "average delivery 31 hours" hides the 10% that take four days.

Common mistakes

• Quoting a mean for skewed money data without the median.
• Reporting a summary of a group with 4 rows next to one with 4,000 - always print count.
• Deleting outliers on sight. Check whether they are data-entry errors (a ₹5,00,000 tea bill) or real business (a genuine bulk order) before touching them.
• Comparing standard deviations across different units or scales instead of the coefficient of variation.
• Averaging percentages or ratios across groups of different sizes. Recompute from the totals instead.