Building an LLM app with Python

Calling a model API is five lines. Shipping something that stays up, stays affordable and can be debugged at 11 pm is the actual work. The pattern below is provider-neutral: model names and vendors change every few months, so nothing above the adapter layer should know which one you use.

One seam for the whole app

Put every provider call behind a single function. Your views, tasks and tests then depend on your interface, not on an SDK.

import os, time, logging, httpx

log = logging.getLogger(__name__)

LLM_BASE_URL = os.environ["LLM_BASE_URL"]      # any OpenAI-compatible endpoint
LLM_API_KEY  = os.environ["LLM_API_KEY"]
LLM_MODEL    = os.environ.get("LLM_MODEL", "default-model")   # names change; keep it in config

class LLMError(RuntimeError):
    pass

def complete(prompt: str, *, system: str = "", max_tokens: int = 500,
             temperature: float = 0.2, timeout: float = 30.0) -> str:
    messages = ([{"role": "system", "content": system}] if system else [])
    messages.append({"role": "user", "content": prompt})
    payload = {"model": LLM_MODEL, "messages": messages,
               "max_tokens": max_tokens, "temperature": temperature}

    for attempt in range(4):
        try:
            started = time.monotonic()
            r = httpx.post(f"{LLM_BASE_URL}/chat/completions", json=payload,
                           headers={"Authorization": f"Bearer {LLM_API_KEY}"}, timeout=timeout)
            if r.status_code in (429, 500, 502, 503, 504):
                raise httpx.HTTPError(f"retryable {r.status_code}")
            r.raise_for_status()
            data = r.json()
            usage = data.get("usage", {})
            log.info("llm ok model=%s in=%s out=%s ms=%d", LLM_MODEL,
                     usage.get("prompt_tokens"), usage.get("completion_tokens"),
                     int((time.monotonic() - started) * 1000))
            return data["choices"][0]["message"]["content"]
        except (httpx.HTTPError, KeyError) as exc:
            if attempt == 3:
                raise LLMError(str(exc)) from exc
            time.sleep(2 ** attempt)          # 1s, 2s, 4s backoff

Swapping to a vendor SDK later means rewriting the body of complete() and nothing else.

Configuration and secrets

Keys live in environment variables or a secret manager — never in code, never in git, never in a prompt. Keep LLM_MODEL, temperature defaults and timeouts in config too, so switching models is a deploy, not a code change.

The four failure modes to plan for

Cheap wins

• Cache deterministic calls by a hash of (model, prompt, settings). Product descriptions and translations repeat.
• Stream long answers so the user sees words in under a second, even if the full reply takes eight.
• Run it in the background — in a Django app, send the request from Celery or a management command, not inside the request-response cycle.
• Log the full trace: prompt id, prompt version, model, tokens, latency, and a truncated output. Debugging blind is impossible.

Test without the network

Because everything goes through complete(), your tests can patch it with a function that returns a canned string. Unit tests then run offline, in milliseconds, and stay stable regardless of which model you ship.

Common mistakes

• No timeout, so one stuck request holds a worker thread until the user gives up.
• Retrying non-retryable errors such as 400 or 401 four times.
• Logging the whole prompt including customer personal data into a third-party log service.
• Treating response["choices"][0] as guaranteed; always handle a missing or empty reply.
• Hard-coding a model name in twelve files and discovering it the day the version is retired.