If you can't test it, you can't ship it

Ordinary software tends to fail loudly. It throws an error, or the page goes blank, or the job grinds to a halt, and somebody files a ticket. Systems built on language models behave differently, and the difference matters more than almost anything else about them. When they go wrong, they usually go wrong quietly: they produce an answer, fluently and with every appearance of confidence, and the answer is subtly or completely wrong. Nothing breaks. The user is simply misled, and depending on the workflow, they may never find out.

That is why we treat testing as the first part of the build rather than the last. Before we write any system, we write the test, and in practice it comes together in three steps.

The first is collecting real questions. We sit with the team who will use the system and gather thirty to fifty questions they actually ask, taken from real emails, calls, tickets and documents. It is tempting to invent tidy examples instead, but invented questions are written by people who already know the answers, and that shows. The awkward, ambiguous, half-formed questions from real life are the ones the system has to survive.

The second is agreeing what a good answer looks like, in writing. Which facts must appear for an answer to count as correct, which source it should cite, and, importantly, when "I don't know" is the right answer rather than a failure. The discipline of writing this down does as much for the project as the testing itself, because it forces a conversation about expectations that most teams have never explicitly had.

The third is scoring every change. A new prompt, a new model, a new body of connected data: run the question set, compare the score, and look at what moved. Anyone who has worked with conventional software will recognise the shape of this. It is regression testing, applied to language, and it exists for the same reason regression testing has always existed: changes that feel safe are sometimes not, and memory is a poor substitute for a measurement.

The score itself turns out to be the less valuable half. What matters more is having it over time. When accuracy can be traced across weeks, you can see whether the system is improving or drifting, you can defend it internally with numbers instead of anecdotes when somebody asks why it is worth the budget, and when something changes upstream (a model is retired, a CRM field is renamed), the test tells you precisely what broke rather than leaving you to rediscover it through complaints. Saying that a system "feels better than it did" is a natural thing to want to say. It is not an engineering result, and sooner or later somebody senior will ask for one.

That is how we build: the test exists before the system does, and it stays in daily use after the system ships. If you'd like to talk about proving an AI system works before your team has to depend on it, get in touch.

Author: Michael Wells