Deploying deep learning models

A model that lives only in a notebook creates no business value. Deployment means packaging the trained weights so another system — a website, a mobile app, a billing counter — can request predictions reliably, quickly and cheaply. This lesson walks through the most common path for Indian startups and SMEs: export, optimise, serve and monitor.

Deployment options at a glance

Option	Best for	Notes
Plain PyTorch + state_dict	Python services	Simplest; needs model code at load time
TorchScript	Python-free C++ runtimes, mobile	Mature; newer projects may prefer torch.export
ONNX + ONNX Runtime	Fast CPU inference, other languages	Portable across frameworks and hardware
TorchServe / Triton	High-volume GPU serving	More setup, batching built in
Managed cloud endpoints	Teams without DevOps	Pay per hour; watch costs in INR

Prepare the model

import torch
from torchvision import models

model = models.resnet18(num_classes=5)
model.load_state_dict(torch.load("sweets_resnet18.pt", map_location="cpu"))
model.eval()
example = torch.rand(1, 3, 224, 224)

Always call eval() before exporting so dropout and batch norm are in inference mode.

TorchScript

scripted = torch.jit.trace(model, example)
scripted.save("sweets_ts.pt")

loaded = torch.jit.load("sweets_ts.pt")
print(loaded(example).shape)   # torch.Size([1, 5])

trace records the operations executed for the example input; models with data-dependent if statements need torch.jit.script instead.

ONNX export

torch.onnx.export(
    model, example, "sweets.onnx",
    input_names=["image"], output_names=["logits"],
    dynamic_axes={"image": {0: "batch"}, "logits": {0: "batch"}},
    opset_version=17,
)

import numpy as np
import onnxruntime as ort

sess = ort.InferenceSession("sweets.onnx", providers=["CPUExecutionProvider"])
logits = sess.run(None, {"image": example.numpy().astype(np.float32)})[0]
print(logits.shape)   # (1, 5)

ONNX Runtime is often 1.5–3x faster than eager PyTorch on CPU, which matters when you are paying for cloud VMs.

Quantisation: smaller and faster

Quantisation stores weights as 8-bit integers instead of 32-bit floats — roughly 4x smaller and often faster on CPU, with a small accuracy drop that you must measure.

import torch.nn as nn
from torch.ao.quantization import quantize_dynamic

text_model = nn.Sequential(nn.Linear(768, 256), nn.ReLU(), nn.Linear(256, 3)).eval()
q_model = quantize_dynamic(text_model, {nn.Linear}, dtype=torch.qint8)

Dynamic quantisation works best for models dominated by linear layers (transformers, LSTMs). CNNs benefit more from static quantisation or from exporting to ONNX and using ONNX Runtime's quantisation tools. Always compare accuracy on your validation set before and after.

Serving with FastAPI

# serve.py  —  run with: uvicorn serve:app --host 0.0.0.0 --port 8000
import io
import numpy as np
import onnxruntime as ort
from fastapi import FastAPI, File, UploadFile
from PIL import Image

CLASSES = ["jalebi", "kaju_katli", "ladoo", "rasgulla", "soan_papdi"]
MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)

app = FastAPI()
sess = ort.InferenceSession("sweets.onnx", providers=["CPUExecutionProvider"])

def preprocess(data: bytes) -> np.ndarray:
    img = Image.open(io.BytesIO(data)).convert("RGB").resize((224, 224))
    arr = (np.asarray(img, dtype=np.float32) / 255.0 - MEAN) / STD
    return arr.transpose(2, 0, 1)[None]          # HWC -> NCHW

@app.post("/predict")
async def predict(file: UploadFile = File(...)):
    logits = sess.run(None, {"image": preprocess(await file.read())})[0][0]
    probs = np.exp(logits - logits.max())
    probs /= probs.sum()
    top = int(probs.argmax())
    return {"label": CLASSES[top], "confidence": round(float(probs[top]), 3)}

The session is created once at startup, not per request. Preprocessing must exactly match training (resize, scale, normalise, channel order); mismatches are the most common deployment bug. For file uploads FastAPI also requires the python-multipart package.

Serving inside Django

If your product is already a Django site, load the model once at module level and call it from a view:

# inference/views.py
from django.http import JsonResponse
from django.views.decorators.http import require_POST
from .model_loader import sess, preprocess, CLASSES   # loaded once per worker

@require_POST
def predict(request):
    upload = request.FILES.get("image")
    if upload is None:
        return JsonResponse({"error": "image is required"}, status=400)
    logits = sess.run(None, {"image": preprocess(upload.read())})[0][0]
    return JsonResponse({"label": CLASSES[int(logits.argmax())]})

For slow models (seconds per request) push work to a task queue such as Celery so web workers stay responsive.

Production checklist

• Version the model file and record which data and code produced it.
• Validate inputs: file type, size limit, image dimensions.
• Log latency, prediction distribution and confidence; alert when they drift.
• Keep a human review path for low-confidence predictions.
• Containerise with Docker so the same environment runs everywhere.