Working with CSV, JSON & pathlib

Most automation work boils down to three steps: find files, read data out of them, write results back. The standard library covers all three without installing anything.

Navigating folders with pathlib

pathlib.Path objects replace string juggling with os.path. The / operator joins paths on every operating system.

from pathlib import Path

base = Path("reports")
base.mkdir(exist_ok=True)

for csv_file in sorted(base.glob("*.csv")):
    print(csv_file.name, csv_file.stem, csv_file.suffix, csv_file.stat().st_size)

out = base / "summary" / "sept.json"
out.parent.mkdir(parents=True, exist_ok=True)

Reading and writing CSV

csv.DictReader turns each row into a dictionary keyed by the header row. Every value arrives as a string, so convert numbers yourself. Always open CSV files with newline="" to avoid blank lines on Windows.

import csv
from pathlib import Path

path = Path("reports/sweets_sales.csv")
path.write_text("item,qty,price\nkaju katli,3,900\nrasgulla,10,40\n", encoding="utf-8")

rows = []
with path.open(newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        row["qty"] = int(row["qty"])
        row["price"] = float(row["price"])
        row["total"] = row["qty"] * row["price"]
        rows.append(row)

with open("reports/sweets_totals.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["item", "qty", "price", "total"])
    writer.writeheader()
    writer.writerows(rows)

Files exported from Excel often start with a byte-order mark. If the first column name looks like \ufeffitem, open with encoding="utf-8-sig".

JSON: loads, dumps, load, dump

JSON maps neatly onto Python: objects become dicts, arrays become lists, true/false/null become True/False/None.

import json

summary = {"shop": "Mithai Ghar", "items": len(rows), "revenue": sum(r["total"] for r in rows)}
Path("reports/summary.json").write_text(json.dumps(summary, indent=2, ensure_ascii=False), encoding="utf-8")

loaded = json.loads(Path("reports/summary.json").read_text(encoding="utf-8"))
print(loaded["revenue"])   # 3100.0

The s versions (loads, dumps) work with strings; load and dump work with open file objects. ensure_ascii=False keeps Hindi or Tamil text readable instead of turning it into \u escapes.

JSON cannot store datetime or Decimal directly. Convert them first, for example with d.isoformat() or str(amount), or pass default=str to json.dumps.

Common mistakes

• Forgetting that CSV values are strings, so "10" + "5" gives "105".
• Omitting encoding="utf-8"; the default on Windows may differ and break non-English text.
• Hard-coding paths like "C:\\Users\\me\\data" — build them with Path.home() / "data" instead.
• Loading a 2 GB CSV into a list. Process rows one at a time inside the with block, or use pandas with chunksize.