Shipping an LLM feature without an evaluation framework is like deploying code without tests — except the failures are often subtle, inconsistent, and visible to your users before your team notices.

Why Most Teams Skip Evals

The typical PM instinct is to eyeball outputs during development, ship, and monitor support tickets. This works for deterministic software. It fails for probabilistic systems where the same input can produce meaningfully different outputs depending on model version, temperature, and prompt drift.

The Four-Layer Eval Stack

A robust evaluation framework for production LLM systems has four layers:

1. Deterministic checks — things that should always be true. Does the response stay within the expected length? Does it contain the required JSON keys? Does it avoid forbidden content categories? These are fast, cheap, and should run on every output.

2. Reference-based scoring — compare outputs against a golden dataset you’ve hand-curated. Use BLEU, ROUGE, or embedding similarity. Best for tasks with a clear correct answer (summarisation, extraction, translation).

3. LLM-as-judge — use a stronger model (GPT-4o, Claude Opus) to evaluate outputs from your production model. Define a rubric: relevance (1–5), completeness (1–5), tone (pass/fail). Surprisingly effective and scales well.

4. Human review sampling — a weekly sample of real production outputs reviewed by subject matter experts. This catches the failure modes that automated evals miss.

Building Your Golden Dataset

Start with 50 examples. Manually review them, keep the ones that represent real production scenarios, discard edge cases. The goal is coverage of your main use cases, not breadth.

Add to it whenever you find a new failure mode in production. Your golden dataset should grow organically, not be designed up front.

Metrics to Track

  • Pass rate on deterministic checks (target: 100%)
  • Mean LLM-judge score week over week
  • Regression rate — what % of previously passing examples now fail after a prompt change
  • Latency P95 — eval quality doesn’t matter if the system is too slow

The PM’s Role in Evals

You don’t need to build the infrastructure. You need to define what good looks like. Write the rubric. Prioritise which failure modes matter most to users. Sign off on the golden dataset. Review the weekly human sample.

That’s it. The engineers handle the machinery. You handle the definition of success.