An agent demo is easy. An agent that runs unattended against production systems, every day, without anyone losing confidence in it, is a different piece of engineering. Most of what matters turns out to be unglamorous, and almost none of it is about picking the right model.
Tool design is the real prompt engineering
Agents fail far more often because a tool was badly described than because the model was not capable. A tool called updateRecord that takes fourteen optional parameters and silently no-ops on a bad enum will produce confident nonsense. The same capability split into three narrow tools with clear names and loud failures will mostly work.
The rule we keep coming back to: a tool should be impossible to call incorrectly without getting an error that explains what to do instead. If a human engineer would need to read the source to use it, an agent will misuse it.
Retries are where the money goes
A failed call is cheap. A failed call that gets retried against a non-idempotent endpoint is a support ticket. We have seen the same category of bug repeatedly: the agent times out waiting for a response, retries, and the downstream system processes both. Now there are two refunds, two tickets, two emails to the customer.
Every tool an agent can call needs an answer to one question. If this runs twice, what happens? If the answer is anything other than nothing, the tool needs an idempotency key before it goes anywhere near an agent loop.
Evaluation beats intuition, but only if it is cheap to run
Everyone agrees evals matter. The teams that actually build them are the ones who made running them take seconds rather than an afternoon. If checking a change requires a person to read fifty transcripts, nobody checks, and the system drifts until a customer notices.
Start with the cases you already argued about. Every time someone says the agent handled something badly, that is an eval case waiting to be written down. A suite of thirty real disagreements is worth more than a thousand synthetic ones.
Escalation is a feature, not a fallback
The most reliable agents we have put into production are the ones that give up early and clearly. They know a short list of situations they are not allowed to handle, and they hand those to a person with the context already assembled.
This is unintuitive if you are measuring automation rate. It makes sense if you are measuring trust. An agent that escalates ten percent of cases cleanly gets used. An agent that attempts everything and is wrong two percent of the time gets turned off, usually within a month, usually after one memorable incident.
What we would tell a team starting now
- Spend the first week on tool surfaces, not on prompts.
- Make every tool idempotent before you make it available.
- Write your first eval case the first time someone complains.
- Define the escalation list before launch, not after the first incident.
- Log the full trace of every run. You will need it, and not at a convenient time.
None of this is exotic. It is ordinary distributed systems discipline applied to a component that happens to be probabilistic. The teams that treat agents as software get software reliability. The teams that treat them as magic get demos.