For most of my career, “done” meant the flow tested well. Five to eight people, a prototype, a list of tasks, a write-up. If people could get through it and understood what happened, we shipped.

With AI features, that’s not enough. You can test a beautiful chat panel with eight people and every session can go fine, simply because the AI happened to give good answers to the eight things they asked. The flow was never the risky part. The answers were.

So I’ve started joining the sessions where the team checks the quality of the AI’s answers. Engineers call this “evals”. I’m not there to fiddle with settings. I’m there because a good set of test questions is a usability test for the part of the product designers usually can’t see.

What I actually do

Early on, I build a small, boring set of test questions. Usually 40 to 80 of them, pulled from support tickets, search logs or interview notes. I write them the way people really write: typos, half-sentences, two questions in one.

Then I agree a short checklist with the team for judging each answer. Four or five points, each a simple yes or no, each tied to something a user would notice:

  • Did it answer the question that was actually asked?
  • Can the claim be checked against a source?
  • Did it pass the person to a human when it should have?
  • Would a user know what to do next?
  • Is the tone right for someone who’s already annoyed?

That last one is a design question, not a technical one. In my experience it’s usually missing until a designer is in the room.

Agree the pass mark first

Two things changed in how I decide something is finished.

First, I set a pass mark before the prototype exists. On one data product, the team agreed up front that a suggestion feature had to get 85% of answers right on a set of 60 test questions. It scored 62%.

  • 60 test questions
  • 85% pass mark agreed up front
  • 62% what the feature scored

Because we’d agreed the bar in advance, killing the feature was a short meeting. Not a month of “it’s nearly there”.

Wrong answers are design input

Second, the failures became design input. When I read through the answers that failed, patterns jump out that no usability session would show me.

The AI is fine on simple questions and falls apart on anything involving dates. It sounds confident about refunds when it should be handing over to a person. Each of those becomes a design decision: a safety rail, a hand-over to a human, a narrower starting point, or a “this isn’t something I can do” message that doesn’t feel like a dead end.

I still run usability tests. They tell me whether people understand the interface. Testing the answers tells me whether what’s inside it deserves their trust. I need both before I’ll call something finished.

The takeaway

If the answer is the experience, the designer should help define what a good answer looks like. A spreadsheet of 60 realistic questions and a five-line checklist is the most useful design artefact I’ve made in the last two years. It doesn’t look like much in a portfolio, which is partly why I’m writing about it.