Iterators, generators & decorators

These three features power much of Python's elegance: for loops, range, file reading, Django's @login_required and functools.lru_cache all rely on them. Understanding them lets you write code that is lazy, reusable and tidy.

The iterator protocol

An iterable is anything you can loop over (list, string, file, dict). Calling iter() on it returns an iterator, an object with a __next__ method that hands out one item at a time and raises StopIteration at the end.

cities = ["Pune", "Nagpur"]
it = iter(cities)
print(next(it))   # Pune
print(next(it))   # Nagpur
# next(it) now raises StopIteration - a for loop catches this for you

An iterator is consumed once. Looping over it a second time gives nothing.

Generators: lazy sequences

A function containing yield becomes a generator. Each yield pauses the function and hands a value to the caller; the next request resumes where it left off. Nothing is computed until someone asks.

def read_large_log(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            if "ERROR" in line:
                yield line.rstrip()

def emi_schedule(principal, annual_rate, months):
    r = annual_rate / 12 / 100
    emi = principal * r * (1 + r) ** months / ((1 + r) ** months - 1)
    balance = principal
    for m in range(1, months + 1):
        interest = balance * r
        balance -= emi - interest
        yield m, round(emi, 2), round(max(balance, 0), 2)

for month, emi, left in emi_schedule(500000, 9.5, 3):
    print(month, emi, left)

A generator expression looks like a list comprehension with round brackets and uses almost no memory:

total = sum(len(line) for line in open("big.txt", encoding="utf-8"))

Decorators: wrapping behaviour

Functions are objects, so a function can take another function and return a new one. A decorator is exactly that, applied with the @ syntax.

import functools
import time

def timed(func):
    @functools.wraps(func)          # keeps the original name and docstring
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.perf_counter() - start:.4f}s")
        return result
    return wrapper

@timed
def slow_total(n):
    return sum(i * i for i in range(n))

slow_total(1_000_000)

@timed above slow_total is shorthand for slow_total = timed(slow_total). The standard library ships useful ones: @functools.lru_cache for memoisation, @property, @staticmethod and @dataclasses.dataclass.

Key takeaways

• Every for loop calls iter() and next() behind the scenes.
• Use generators for large files, streams and infinite sequences; they keep memory flat.
• Generators are single-use: convert with list() if you need to loop twice.
• Always use functools.wraps inside decorators, and pass through *args, **kwargs and the return value.