You will not see a bad AI demo. Not because the products are uniformly good, but because a demo is a curated path through a curated dataset, and every vendor has had months to find the path that works. The useful question is not whether the demo is impressive. It is what the vendor is prepared to show you that they did not choose.
Ask for your data, before the contract
The single most informative request is to run the system against a sample you select. Not a sample they request. Not a sanitised extract. A slice of real cases that you picked, including the awkward ones.
Vendors who can do this will do it quickly and it will tell you a great deal. Vendors who cannot will explain that a proper evaluation requires a longer engagement. That is sometimes legitimate and it is always worth noticing, because it means you are being asked to buy on the strength of their data rather than yours.
Make them name the failure modes
Every system has a shape to how it fails. Ask directly: what kinds of cases does this get wrong, and how would we know? A vendor with production customers can answer specifically. They will tell you about the format that confuses the parser, or the category of request that needs a human, because they have been fixing it for two years.
A vendor who says it does not really fail is telling you they have not watched it closely, or they are not going to. Both are useful to know before you sign.
Look at the tail, not the average
Accuracy numbers are averages, and averages hide the thing that will hurt you. Ninety-four percent sounds strong until you ask what the other six percent looks like. If the failures are near misses that a reviewer catches, that is a workable system. If some of them are confident, plausible and wrong, that is a different product with a different risk profile.
Ask to see the worst twenty outputs from a recent week. The answer, including whether they can produce it at all, tells you how the system is operated.
Price the exit while you still have leverage
Before signing, establish what leaving looks like. Can you export the data, including anything derived? Do you keep the evaluation cases you built? What happens to fine-tuned artefacts trained on your material? If the answers are unclear at the start, they will be unfavourable at renewal.
Five questions to take into the next meeting
- Can you run this on fifty cases we choose, this week?
- What does it get wrong, and how would we find out?
- Show us the worst twenty outputs from a recent week.
- What do we take with us if we leave, including evals and tuned models?
- Who at your company gets paged when this is wrong for a customer?
None of these are hostile and a good vendor will enjoy them. The ones who struggle are usually struggling for a reason that would have become your problem a year in.