You changed a tool. The new output looks better. Did it get better?

You do not know yet, because you looked at 1 example. It was the one you had open, and it was the case that made you edit in the first place, so it is also the case the edit was tuned to fix.

This is how prompt work usually goes wrong. A run of sensible edits, each one checked against the case that prompted it, that together make the tool worse on everything else.

The smallest thing that counts

5 saved inputs. That is the method.

A tool takes 1 input, so a test case is 1 block of text, 1 link or 1 file. Keep 5 of them in a note somewhere and run both versions of the tool against all 5 before you ship a change.

Picking them is the part that matters:

  1. The typical case. The one the tool exists for.
  2. The case that made you edit. The reason you are here.
  3. The shortest plausible input. A 2-line email. An empty section.
  4. The longest plausible input. The 40-page PDF, the 3-hour recording. Something will get dropped, and you want to know what.
  5. The one that is slightly wrong. A missing field, the wrong language, formatting mangled on the way in.

Cases 3 to 5 are where regressions live. Case 1 almost never changes. Case 2 always improves, which is why you made the change.

Read the difference between the two

Run both versions and put the 2 results side by side. That is the whole technique, and it works because your eye is much better at spotting a change than at judging quality on its own.

Look for 3 things:

  • What did the new version stop doing?
  • What did it start doing that nobody asked for?
  • Did the shape change? A heading that vanished, a field that used to be there.

The second one catches most of the damage. Models are agreeable. Add an instruction to fix one thing and you often get behaviour nobody requested along with it.

Write the expectation down first

2 sentences, before you run anything:

Change: told it to always include the order number.
Expect: inputs 1, 2 and 5 gain an order-number line. 3 and 4 unchanged.

Then run it. Anything outside that prediction is a side effect, and side effects are what you are hunting.

This costs 30 seconds and turns "the output looks good" into a claim that can be wrong. Over a few months the file of predictions also becomes the most honest document you have about how the tool behaves.

The Test button in the builder runs the draft you are looking at without saving it and without recording a run, so you can do this on an edit you have not saved yet.

When the input is longer than the model can read

One failure deserves its own name, because it is invisible.

Feed a model a transcript longer than its context window and it will not tell you it ran out. It answers from the part it saw, confidently. If your 5 inputs are all short, you will never find this. The day somebody runs the tool on a 5-hour recording, they get a summary of the first hour presented as a summary of the whole thing.

That is what case 4 is for. Every tool here is told to say when the input exceeded what it could read, and to say so up front rather than answering as if it had the whole thing. Check that yours does.

The rule underneath all of it

Keep the inputs. Run them before and after. Read the differences.

Everything more sophisticated (scoring, model-graded rubrics, regression suites in CI) is a refinement of that loop, and none of it helps if you skip the loop.

Related: from a prompt to a tool your team can use and why every edit needs a snapshot.