Capstone: sweet image classifier end to end

This capstone ties the whole track together. You will build MithaiVision: an app that lets staff at a sweet shop photograph a tray and get the sweet's name and price per kg, useful for billing counters staffed by new employees during the festive rush. The same steps apply to any image project — apparel categories, spare parts, crop diseases.

1. Define the problem and success

Item	Decision
Task	Multi-class image classification
Classes	8 sweets: jalebi, kaju katli, gulab jamun, rasgulla, ladoo, barfi, soan papdi, peda
Metric	Macro F1 (every class matters equally) plus top-1 accuracy
Target	Macro F1 of at least 0.90 on the held-out test set
Latency	Under 300 ms per image on a 2-vCPU server

Writing the target down before you start prevents endless tinkering.

2. Collect and label data

• Photograph 150–300 images per class across different trays, lighting, angles and phones.
• Include "hard" cases: half-empty trays, silver varq, reflections from glass counters.
• Split by photo session or day, not randomly, so near-duplicate shots do not appear in both train and test. Aim for 70/15/15.

mithai/
  train/<class>/*.jpg
  val/<class>/*.jpg
  test/<class>/*.jpg

Keep a labels_audit.csv noting doubtful images; ambiguous labels (peda vs barfi) cap your accuracy more than any model choice.

3. Build the pipeline and a baseline

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets, models, transforms

weights = models.EfficientNet_B0_Weights.DEFAULT
norm = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
train_tfms = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.6, 1.0)),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(15),
    transforms.ColorJitter(0.2, 0.2, 0.1),
    transforms.ToTensor(), norm,
])
eval_tfms = transforms.Compose([
    transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), norm,
])

train_ds = datasets.ImageFolder("mithai/train", train_tfms)
val_ds = datasets.ImageFolder("mithai/val", eval_tfms)
train_dl = DataLoader(train_ds, batch_size=32, shuffle=True, num_workers=4)
val_dl = DataLoader(val_ds, batch_size=64, num_workers=4)

model = models.efficientnet_b0(weights=weights)
for p in model.features.parameters():
    p.requires_grad = False
model.classifier[1] = nn.Linear(model.classifier[1].in_features, len(train_ds.classes))

Train only the head for about 5 epochs with AdamW (lr 1e-3) and CrossEntropyLoss(label_smoothing=0.1), using the fit() loop from the training-loop lesson. This is your baseline.

4. Improve systematically

Change one thing at a time and log every run (a spreadsheet or a tool like MLflow is fine):

1. Unfreeze the last two feature blocks (model.features[-2:]) with lr 1e-4, head at 1e-3.
2. Add early stopping (patience 5) and keep the best checkpoint.
3. If one class lags, inspect its errors and collect more photos rather than adding layers.
4. Try a bigger backbone only if you are still short of the target.

5. Evaluate honestly

Run the test set once, after all choices are frozen:

from sklearn.metrics import classification_report, confusion_matrix

@torch.no_grad()
def predict_all(model, dl):
    model.eval()
    ys, ps = [], []
    for xb, yb in dl:
        ps.append(model(xb).argmax(1))
        ys.append(yb)
    return torch.cat(ys).numpy(), torch.cat(ps).numpy()

test_ds = datasets.ImageFolder("mithai/test", eval_tfms)
y_true, y_pred = predict_all(model, DataLoader(test_ds, batch_size=64))
print(classification_report(y_true, y_pred, target_names=test_ds.classes, digits=3))
print(confusion_matrix(y_true, y_pred))

Study the confusion matrix: if barfi and peda are confused, show staff both suggestions whenever confidence is below a threshold, say 0.6.

6. Deploy

1. Export the best model to ONNX with a dynamic batch axis.
2. Serve it with the FastAPI app from the deployment lesson; add a price lookup:

PRICE_PER_KG = {"jalebi": 480, "kaju_katli": 1200, "gulab_jamun": 520, "rasgulla": 500,
                "ladoo": 560, "barfi": 800, "soan_papdi": 450, "peda": 700}

3. Build a simple Django or mobile page with a camera upload button that shows the label, confidence and price.
4. Containerise with Docker and deploy to a small cloud VM.

7. Monitor and iterate

• Log each prediction, confidence and whether staff accepted or corrected it.
• Corrections become new labelled data; retrain monthly or before big festivals.
• Watch for drift: a new packaging style or a new sweet on the menu requires fresh data.

Alternative capstone: demand forecasting

If you prefer time series, forecast next-week demand per SKU for a chain of stores: build 28-day windows with calendar and festival features, compare seasonal-naive, LightGBM and GRU models on a time-based test period, and serve weekly forecasts as a CSV for the purchase team. The same discipline — clear metric, honest splits, baseline first, monitoring — applies.

Deliverables checklist

• Problem statement with metric and target
• Data card: sources, class counts, split method, known issues
• Experiment log with at least five runs
• Test-set report and confusion matrix
• Working API plus a short demo video
• A one-page README explaining limitations