For AI engineering
Eval a prompt on a real model
Edit a ticket classifier's prompt, run it against a ten-case golden set, and see the score and regressions change.
What to take away
- An eval is a fixed set of inputs with known answers. Run it before and after every prompt or model change.
- Structured outputs make grading a plain comparison in code. Use a model as judge only when there's no single right answer.
- Most misses come from labels nobody defined. Write the definitions the customer's team already uses.
- Watch for regressions, not just the score: a change that fixes two cases and breaks one is worth knowing about.
- Ten cases teach the idea. A real golden set comes from real tickets, including the ugly ones, and grows with every bug report.