Where to start when evaluating any generative AI surface

Every team shipping a generative AI feature eventually asks the same question: is it any good? The honest answer, for most teams, is that they do not know. The model demos well. The metrics dashboard is green. Then a customer hits a failure mode nobody measured, and the team discovers that “quality: 4.2/5” was measuring nothing at all.

I own model quality at Photoroom, where generative models touch product photos for e-commerce sellers 300 million downloads deep. Before that I shipped neural networks that read cardiograms in clinical production. In both jobs the model was the easy part. Knowing whether it worked was the hard part. This essay is the playbook I wish someone had handed me.

What is eval-driven development?

Eval-driven development is test-driven development for systems that have no single correct answer. Instead of predicting every failure up front, you build the habit and the infrastructure to discover failures, encode them as evaluations, and run those evaluations on every change. The evals become the specification. They force the team to say, in writing, what “good” means.

This matters because generative AI breaks the assumptions of software testing. Outputs are non-deterministic. Correctness is subjective. And for images, the break is total: there is no schema to validate, no string to match, no assertion to write. Code cannot tell a faithful product photo from a subtly ruined one. You cannot assert-equals your way out of that.

Rule one: read your data, yourself

The highest-return activity in AI evaluation costs nothing and requires no tooling. It also does not scale, which is precisely the point.

At Photoroom, I review image generations personally. Not aggregate scores, not a dashboard, the images themselves, one by one. It is how I understand what the models actually do to our users’ products. Dashboards tell you a metric moved. Looking tells you why. The interesting failures are never “bad image”. They are specific and nameable: a warped logo, an invented texture, lighting that flatters the model and betrays the product. You cannot evaluate what you have not named, and you cannot name what you have not seen.

So start manual, and stay manual longer than feels comfortable. Run a hundred inputs through your prototype and read every output before you build anything else. Startups are told to do things that don’t scale to find product-market fit. The same advice applies to quality: the scaled pipeline comes later, and it will only be as good as the taxonomy of failures you built by hand.

Evaluating images: humans first, then your own raters

Because code cannot judge an image, evaluating image generation means building raters: models that score outputs against a criterion. The question is whose raters, and the answer took us some trial and error.

Off-the-shelf raters are not there yet. We tried them. Aesthetic scorers and general-purpose vision models can tell a good image from a catastrophe. They cannot apply a precise criterion like fidelity: whether the generated photo still shows the product the seller is actually selling. A rater that cannot see a warped logo is not a fidelity rater, whatever its benchmark scores say. Generic raters measure generic quality. Your product does not compete on generic quality.

That leaves two conclusions, and they are the core of this essay.

First, for the criteria that matter, you must go through humans. There is no shortcut. When we made product fidelity a company bet, the real work was not infrastructure. It was turning “it’s not realistic” into criteria people could agree on, then ranking failures by severity with trained annotators. Human disagreement is a definition problem, and no model fixes a definition problem. Sometimes the humans can even be the public: we ran a blind arena on Hugging Face where anyone could vote between four background removers. Nine thousand votes produced an Elo ranking and a defensible answer to “are we actually the best at this?” Votes at scale are the closest thing generative AI has to ground truth.

Second, human labels are measurements and training data at once. Every annotation does double duty. It tells you how good the model is today, and it teaches a future rater what to look for. Once annotators have applied your criteria to enough examples, you train your own raters on those labels: models built for the specific criteria that matter to Photoroom, not for a benchmark’s idea of quality. This is the crux. Collect the data yourself, on the dimensions your product competes on, and build raters that encode your definition of good rather than someone else’s. The humans set the standard; the rater industrialises it.

The same lesson, translated to LLMs

Everything above transfers to text, almost line for line.

The generic LLM-as-a-judge is the off-the-shelf rater of the text world. Prompt a strong model to score “helpfulness” and it will produce confident numbers that correlate with nothing your users care about. It fails for the same reason aesthetic scorers fail on fidelity: your product’s precise criteria, faithfulness to your policies, your tone, your domain’s definition of correct, are not in anyone’s pretraining.

The sequence is the same too. Humans first: have the people who know the domain label 50 to 100 examples, failures included, and if they disagree with each other, stop and fix the definition before automating anything. Then build your own judge: one judge per dimension, each with a rubric a human could apply consistently, because if a human cannot apply it, a model cannot either. Then calibrate: measure agreement between judge and human labels, aim for the high 80s in percent, study the disagreements, tighten the rubric, repeat. An uncalibrated judge is a random number generator with a good vocabulary. A calibrated one is your annotators, industrialised. And as with images, the labels do double duty: today’s measurement is tomorrow’s training data for a judge built on your criteria.

One addition for agentic systems: score the trajectory, not just the answer. An agent can reach the right output through the wrong tool call or a lucky guess that will not repeat. Persist the traces and evaluate the path as well as the destination.

Evaluation does not stop at launch

Pre-launch evals are a snapshot. Models drift, prompts change, users find inputs you never imagined. So mirror the stack in production: sample a slice of live traffic daily, de-identified; run your raters on it; surface flagged outputs for human review; and keep reading the data yourself. The manual habit that started the loop is also what keeps it honest. Every reviewed output feeds the label pool, and the label pool feeds the next generation of raters. The loop never closes because the product never stops moving.

A checklist for your first eval stack

1. Run 100 inputs through the prototype. Read every output yourself.
2. Name the failure modes you find. Specific names, not “quality issues”.
3. Try off-the-shelf raters, but test them against your own judgment before trusting them. Expect them to fall short on precise criteria.
4. For the criteria your product competes on, define them with humans first.
5. If your experts disagree with each other, fix the definition before automating.
6. Label 50 to 100 examples, bad ones included. This is your golden set.
7. Treat every label as double duty: measurement today, training data tomorrow.
8. Train or calibrate your own raters on that data, one per criterion.
9. Trust a rater only once it agrees with your annotators in the high 80s percent.
10. Mirror the whole stack in production, and keep looking at outputs yourself.

FAQ

Why not just use an off-the-shelf evaluator?

Because generic raters measure generic quality. They can separate good from catastrophic, but they cannot apply precise criteria like product fidelity or faithfulness to your policies. For the dimensions your product competes on, you need humans to define the standard, then your own raters trained on their labels.

Why can’t you evaluate generated images with code?

There is nothing for code to check. An image has no schema to validate or string to match, and the failures that matter, a warped logo, an invented texture, are judgments, not assertions. Evaluating images means building raters, and raters worth trusting are trained on human labels.

How many examples do you need to start evaluating a generative AI feature?

About 100 to read manually, and 50 to 100 expert-labeled examples for a golden dataset. Small and sharp beats large and noisy. Scale the dataset only once the criteria are stable and volume is the bottleneck.

Can an LLM reliably evaluate another LLM?

Only after calibration against human labels, and only on a well-defined dimension. Target agreement with your annotators in the high 80s to 90s percent. An uncalibrated judge gives false confidence, which is worse than no judge.

What is a golden dataset?

A fixed set of inputs with expert-approved labels, used to test the evaluators themselves. It must contain failures as well as successes, otherwise it cannot measure whether your rater can tell them apart. It is also the seed of your training data for custom raters.

How much effort should evaluation take?

A meaningful share of the total project, including senior time spent looking at raw outputs. It is not overhead. It is the difference between a product that works and a demo that impressed the room once.