Tokens, context windows & cost

Every LLM API bills you, limits you and slows down in units of tokens. Understanding tokens is the difference between a prototype that costs INR 500 a month and one that quietly burns INR 5 lakh.

What a token is

A tokenizer splits text into sub-word pieces from a fixed vocabulary. Common English words are often one token; rare words, code and many Indian-language scripts split into several. A rough rule for English is about 4 characters or 0.75 words per token, but Hindi or Tamil text can take 2 to 4 times as many tokens for the same meaning. Always measure with the tokenizer your provider documents rather than guessing.

# pip install tiktoken  (an open-source tokenizer used by several model families)
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
for text in ["Invoice total including GST", "कुल राशि जीएसटी सहित"]:
    print(len(text), "chars ->", len(enc.encode(text)), "tokens")

Different model families use different tokenizers, so treat any count from another tokenizer as an estimate.

The context window

The context window is the maximum number of tokens the model can consider in one request: system prompt + conversation history + retrieved documents + tool definitions + the output it generates. If you exceed it, the API returns an error or older content gets truncated. Even within the limit, very long prompts are slower and the model can overlook details buried in the middle.

Strategies when you run out of room:

• Summarise older chat turns and keep only the recent ones verbatim.
• Retrieve only the top few relevant chunks (RAG) instead of pasting whole PDFs.
• Set a sensible maximum output length for each task.

Estimating cost

Providers typically charge per million tokens, with output tokens priced higher than input tokens. Some also discount repeated prompt prefixes (prompt caching) or offer cheaper batch processing. Prices and model names change frequently, so keep them in configuration, not in code.

def monthly_cost_inr(requests_per_day, in_tokens, out_tokens,
                     in_price_per_m, out_price_per_m, usd_to_inr=84):
    per_req_usd = (in_tokens * in_price_per_m + out_tokens * out_price_per_m) / 1_000_000
    return round(per_req_usd * requests_per_day * 30 * usd_to_inr)

# Example: support bot, 3,000 chats/day, 1,800 input + 300 output tokens each,
# at illustrative prices of USD 0.50 / 1.50 per million tokens
print(monthly_cost_inr(3000, 1800, 300, 0.50, 1.50))   # 10206

Each request costs USD 0.00135, so 3,000 x 30 requests come to about INR 10,200 a month. Double the history you re-send, or switch to a model priced ten times higher, and the bill changes just as fast.

Common mistakes

• Forgetting that chat history is re-sent on every turn, so costs grow with conversation length.
• Estimating tokens from word counts for non-English text.
• Hard-coding a model name and price; when the provider retires the model, your app breaks.
• Not logging token usage per request; the API response usually reports input and output token counts.