OCR on Indian documents: what Tesseract gets right and where it fails

Tesseract handles a straight 300 DPI scan of laser-printed English well enough that you will be fixing a word or two per page. Give it a Devanagari notice and it still does useful work, with matras and conjuncts as the weak point. Give it a table and it hands back words in an order that looks random. Give it handwriting and it produces nonsense. Almost every accuracy problem blamed on the engine is really an input problem, which is good news: input is the part you control.

What the engine actually does

Tesseract 4 and later is an LSTM line recogniser. The pipeline: grayscale and threshold to black-and-white, find connected components, group them into text lines and a reading order, feed each line to the neural net one column of pixels at a time, decode the characters with help from a language model built from the training text.

Two consequences matter. Everything downstream depends on that thresholding step, so a shadow across the page removes text before recognition begins. And the language model is why Invoice comes out right while a part number like X4R-90ZQ gets mangled: strings outside normal vocabulary get no help.

Prepare the image, then complain about accuracy

Roughly in order of how much they matter:

• Resolution. Aim for text 25 to 40 pixels tall, which a 300 DPI scan of 10 pt text gives you. Below about 150 DPI accuracy collapses; 600 DPI rarely helps and doubles the time.
• Straightness. More than 2 degrees of skew starts merging lines.
• Contrast, evenly. A phone photo under a ceiling light has a bright side and a dark side, and global thresholding wipes out the dark half. Use an adaptive threshold.
• Cropping. Remove the desk, the fingers and the black scan border; page-edge noise confuses layout analysis.
• No rescue JPEGs. A photo saved at quality 40 has ringing around every letter that re-saving will not remove.

import cv2, numpy as np

def prep(path):
    img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
    if img.shape[0] < 1200:                       # upscale small crops
        img = cv2.resize(img, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)
    angle = cv2.minAreaRect(np.column_stack(np.where(img < 200)))[-1]
    if angle > 45:
        angle -= 90
    if abs(angle) > 0.5:                          # deskew
        h, w = img.shape
        m = cv2.getRotationMatrix2D((w // 2, h // 2), angle, 1.0)
        img = cv2.warpAffine(img, m, (w, h), flags=cv2.INTER_CUBIC, borderValue=255)
    img = cv2.bilateralFilter(img, 7, 50, 50)     # denoise, keep edges
    return cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                 cv2.THRESH_BINARY, 31, 15)

The two numbers worth tuning are the last pair. Block size 31 with a constant of 15 suits a photographed A4 page; for a dense printed rate chart try 15 and 8. Preview the binary image before tuning anything else, because if the text looks broken to you it will look broken to the recogniser.

Devanagari and the other Indic scripts

Install the language data and name it explicitly. pytesseract.image_to_string(img, lang="hin+eng") is the right call for a Hindi document with English numerals and product codes, which is most Indian paperwork. Put the dominant script first. The codes are hin, mar, san, ben, tam, tel, kan, mal, guj, pan, ori, urd, plus a script-level Devanagari model worth trying when the language is mixed.

Where Devanagari goes wrong is structural. The shirorekha, the horizontal line joining letters across a word, gives line-finding a strong horizontal feature, and where ink has spread, adjacent characters merge under it. Matras above and below the line get dropped or attached to the wrong consonant, so कि and की swap, and half-forms in a conjunct come back as two letters. The text stays readable to a human and fails an exact-match check, which matters when you are looking names up against a database.

Use tessdata_best rather than the default tessdata_fast for Indic work. It is three to four times larger and slower, and the difference on Devanagari is obvious on the first page. Urdu is harder again, right-to-left with heavy ligatures, and poor enough that I would not build a process on it.

Tables

Tesseract has no idea what a table is. It finds words and lines, so a bordered two-column table often returns one line with the left and right cells stuck together, or the whole first column followed by the whole second. --psm 6 treats the image as one uniform block and stops it inventing a column layout. The reliable approach is to stop asking for text and ask for positions:

import pytesseract, pandas as pd

d = pytesseract.image_to_data(img, lang="eng", config="--psm 6",
                              output_type=pytesseract.Output.DATAFRAME)
d = d[(d.conf > 40) & d.text.notna() & (d.text.str.strip() != "")]
d["row"] = (d.top // 18)                       # bucket by line height
table = (d.sort_values(["row", "left"])
           .groupby("row")["text"].apply(list))

Cluster words by top into rows, sort each row by left, and you have the grid. The bucket size has to come from your own document; 18 pixels suits body text on a 300 DPI scan and will merge two rows on a tightly printed statement. Our Image to Excel tool does this clustering in the browser and lets you correct cells before downloading the .xlsx, which beats tuning thresholds for a one-off sheet.

Handwriting: the honest answer

Tesseract cannot read handwriting. It was trained on printed glyph shapes, and cursive or even neat handwritten Devanagari is a different recognition problem: you will get output, it will be words, and almost none will be the right words. Block capitals written inside form boxes sometimes work, which is why some forms are designed that way. Cloud handwriting models do markedly better on English, and for handwritten Hindi or regional scripts none I have tested are good enough to run without a human checking every field.

Confidence scores, and what they do not tell you

image_to_data returns a conf value from 0 to 100 per word, with -1 for layout elements containing no text. A mean above roughly 85 usually means a clean page; below 60, fix the image.

Treat it as triage, not an accuracy figure. Confidence says how sure the network was about the characters it emitted, not whether the reading order is right or a column was missed. I have seen a page average 92 while dropping the last line of every paragraph. Use low confidence to flag pages for review, never to claim a document was read correctly.

Any percentage figure quoted for OCR accuracy is meaningless without the input it was measured on, so measure your own. Take twenty representative pages, count the fields you had to correct, divide. That number is what your process must survive, and it changes the day somebody replaces the scanner.

Repairing the output with regex

For invoices and ID documents you rarely need the whole text, you need six fields. Extract them with patterns and repair the known confusions first, because 0 reads as O, 1 as I or l, 5 as S and 8 as B constantly.

import re

PAN = re.compile(r"\b([A-Z]{5}[0-9]{4}[A-Z])\b")
GSTIN = re.compile(r"\b(\d{2}[A-Z]{5}\d{4}[A-Z][A-Z\d]Z[A-Z\d])\b")
DATE = re.compile(r"\b(\d{1,2})[/\-.](\d{1,2})[/\-.](\d{2,4})\b")
AMOUNT = re.compile(r"(?:Rs\.?|INR|\u20b9)\s*([\d,]+(?:\.\d{2})?)")

DIGITS = str.maketrans({"O": "0", "o": "0", "I": "1", "l": "1", "S": "5", "B": "8"})
LETTERS = str.maketrans({"0": "O", "1": "I", "5": "S", "8": "B"})

def fix_pan(token):
    t = token.upper().replace(" ", "")
    if len(t) != 10:
        return None
    t = t[:5].translate(LETTERS) + t[5:9].translate(DIGITS) + t[9:].translate(LETTERS)
    return t if PAN.fullmatch(t) else None

The pattern is: find the candidate loosely, coerce each position to the character class the format demands, then validate. A GSTIN is 15 characters with the state code first and Z in position 14, so almost every OCR slip in one is correctable without guessing. Dates need a sanity check on day and month, plus an assumption written down that 04/03/2026 is 4 March, because Indian documents are day-first and the regex cannot tell.

Amounts need care: OCR drops decimal points and commas more often than digits, so 1,250.00 becomes 125000. Cross-check the total against the line items and flag a mismatch rather than trusting either.

A worked example

A supplier sends a photographed rate card: one page, twelve rows, Hindi item names and English prices.

1. Crop to the page and check it is not tilted more than a degree or two.
2. Run it through AI OCR with English and Hindi selected and enhance left on. This two-minute check tells you from the average confidence whether the photo is good enough, before you write any code.
3. If confidence is low, retake the photo in daylight instead of fighting it in software. That beats every filter above.
4. Feed the same image to Image to Excel for the grid, fix the wrong cells, download the spreadsheet.
5. Run the amount regex over the price column and compare the total against the printed one.

What failed on the last card I did this with: two Hindi item names lost a matra, one price read 1,B50, and a merged header cell became a row of its own. All three were faster to fix in the grid than to solve with preprocessing. That is the realistic shape of an OCR project. The engine gets you close and a person closes the gap, so the engineering goes into making corrections quick and into never letting an unchecked field reach a ledger.

Where volume makes checking unworkable the fix is a validation layer, not a better engine, which is what our automation work builds; the pandas track covers cleaning the extracted output.

Frequently asked questions

Should I use Tesseract or a cloud OCR API? Tesseract when the documents are printed, volume is moderate, and the files should not leave the device. A cloud API when you need handwriting, multi-column layouts or table structure returned for you, and the documents are not sensitive enough to make sending them a problem.

Why does the same image give different results in my script and in the browser tool? Different preprocessing, and often a different model build. The browser tool upscales and sharpens first; your script might not. Match the preprocessing before concluding one engine is better.

Is 100 percent accuracy possible? No, and any tool claiming it is not measuring honestly. Design the process so a wrong field is caught by a validation rule or a human, not so it never happens.