Client Portal Get a Quote
python

HTTP requests with retries & timeout

Robust API calls using requests + urllib3 Retry.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(total=5, backoff_factor=0.5, status_forcelist=[429, 500, 502, 503, 504])
session.mount("https://", HTTPAdapter(max_retries=retry))

resp = session.get("https://api.example.com/items", timeout=10)
resp.raise_for_status()
print(resp.json())

Why plain requests.get() isn't enough

APIs fail. Not often, but often enough: a payment gateway times out, a government portal returns a 503 during peak hours, an API sends 429 "too many requests". A script that calls requests.get(url) once just crashes, usually at 2 a.m. when nobody is watching.

How this works

The Session reuses the same connection for every request, which is faster when you call the same API many times.

Retry says what to do when something goes wrong:

  • total=5: try up to 5 more times before giving up.
  • status_forcelist=[429, 500, 502, 503, 504]: only retry on these, the "try again later" kind of errors. A 404 or 401 won't fix itself, so there's no point retrying those.
  • backoff_factor=0.5: wait a bit longer between each try, roughly 0.5s, 1s, 2s, 4s and so on. This gives a struggling server time to recover instead of hammering it.

timeout=10 is the line people forget most. Without a timeout, requests can wait forever for a server that never replies, and your script just hangs.

raise_for_status() turns a 4xx or 5xx reply into an exception, so errors don't slip through looking like normal data.

Things to watch

  • By default, Retry doesn't retry POST requests, because sending a payment or an order twice could be a disaster. Only allow it if the API says repeated POSTs are safe.
  • If the server sends a Retry-After header with a 429 or 503, urllib3 waits as long as it asks.
  • Mount the adapter on "http://" too if you call any non-HTTPS URLs.