Evaluating LLM apps: hallucinations, RAG or fine-tuning

"It looked good when I tried it" is not an evaluation. LLM outputs vary, prompts change and models get upgraded, so you need a repeatable way to know whether your app got better or worse.

Build an evaluation set first

Collect 50 to 200 realistic inputs with expected outcomes, including tricky ones: questions your documents cannot answer, ambiguous queries and inputs in Hinglish. Store them in a file and run them on every prompt or model change, just like unit tests.

import json

cases = [json.loads(line) for line in open("eval/hr_policy_qa.jsonl", encoding="utf-8")]
# each line: {"question": ..., "must_include": ["12 days"], "should_refuse": false}

def grade(case, answer):
    if case["should_refuse"]:
        return "not in the policy" in answer.lower()
    return all(s.lower() in answer.lower() for s in case["must_include"])

results = [grade(c, ask_bot(c["question"])) for c in cases]
print(f"pass rate: {sum(results) / len(results):.0%}")

Types of checks

LLM judges are useful but biased (they may prefer longer answers), so spot-check them against human ratings.

Hallucinations

A hallucination is fluent, confident text that is not supported by facts or by the provided context. Reduce it by:

• Grounding answers in retrieved documents and asking the model to cite the chunk it used.
• Explicitly allowing "I don't know" in the system prompt, and testing that it happens.
• Using low temperature for factual tasks.
• Checking faithfulness: does every claim in the answer appear in the context?
• Keeping facts that must be exact (prices, stock, balances) in tools or databases, not in the model's memory.

Prompting vs RAG vs fine-tuning

Fine-tuning teaches behaviour, not reliable facts; a fine-tuned model will still hallucinate facts about your latest price list. The usual order is: better prompts first, then RAG for knowledge, then fine-tuning only when evaluation shows a gap that the first two cannot close. They also combine well: a fine-tuned small model can answer from RAG context.

Common mistakes

• Changing the prompt and the model at the same time, so you cannot tell which helped.
• Evaluating only on happy-path questions.
• Fine-tuning to "add knowledge" that changes weekly.
• Not tracking cost and latency alongside quality.