Questions to ask an AI agency before you sign

Ask an AI agency eight things before you sign: what it shipped that is still running, how often a human steps in, how “done” will be tested, who does the work, what the system costs to run after handover, how it handles hostile input, what you receive if you leave, and what it will refuse to build. Each answer should be a document or a demo, not a reassurance. An agency that can only answer in adjectives has told you what you needed to know.

We sell on the other side of this table, so we have heard the questions that matter and the ones that do not. This list is what we would bring if we were the buyer.

Why the questions have to be harder than for ordinary software

AI vendors are selling a capability that is easy to claim and hard to see. Regulators have started to notice. On 14 January 2025 the US Securities and Exchange Commission settled with Presto Automation over its AI drive-through ordering product. The order found that Presto disclosed in November 2023 that over 70% of orders taken by its voice product required human agent intervention, while the “without intervention” rate it had advertised counted only staff inside the restaurant.

On 25 August 2025 the Federal Trade Commission sued Air AI, alleging it told small businesses its agents could replace human customer service reps and promised refunds to customers who missed earnings targets, then rarely honored the guarantee. The FTC said some customers lost as much as $250,000. In March 2026 the company settled and was banned from marketing business opportunities.

Neither case is typical. Both are what the gap between a claim and a system looks like when someone finally checks. Your checking happens before the contract.

1. What did you ship that is still running?

The question to ask is: “Show me a system you built that is in production today, and tell me who operates it.” A good answer is a named, working system, a live walkthrough, and a client or internal owner you can ask. A portfolio of screenshots and logos is a weak answer, because a screenshot can come from a prototype that never ran twice.

Ask how long it has been running and what has broken. Every production system has failure history. A builder who cannot name a single failure either has not run anything or is not telling you.

For reference, our own companies run 41 agents in production as of July 2026, each a tomte, our word for one production agent with one defined job. The agent fleet proof page lists what they do.

2. How often does a human step in, and how is that measured?

Ask for the share of runs that finish without a person touching them, and ask what counts as a touch. The Presto order is the cautionary tale: one metric, two definitions, and a gap wide enough to reach an SEC order.

Human review is not a flaw. Many good systems keep a person on the risky steps on purpose. The flaw is a number nobody can define. Ask for the denominator, the period it covers, and whether off-site or back-office handling counts. Then ask what the system does when it is unsure: stop, flag, or guess.

3. How will “done” be tested?

Done should be a written test on your real inputs, agreed before the build. The contract mechanics are in our guide to AI development contract terms, so here is only the question: “What are the pass criteria, who runs the test, and what happens if it fails?”

Look for a specific count of sample cases and a stated pass rate. “We’ll iterate until you’re happy” means the scope is whatever the builder says it is on the last day.

4. Who actually does the work?

Ask who writes the code, who reviews it, and whether any of it is subcontracted. Agencies subcontract all the time and that is fine, but the person signing your contract should be able to name who is doing what, and the subcontractor should be bound by the same terms on your data and your code.

Ask who you talk to when something breaks at a bad hour, and in what channel. If the answer depends on a weekly meeting, you are buying meetings. We run on written updates and a reply inside one business day, which is also why we price without a call.

5. What does it cost to run after you leave?

An AI system has a running cost a website does not: model usage is billed per call. Ask for a monthly estimate at your expected volume, whose account the API key sits on, and who pays when the model vendor retires the model underneath you.

If the agency holds the key, the bill reaches you through their invoice, with a margin, and you cannot see per-run cost. If you hold it, you see every cent and can rotate it the day the engagement ends. The model-retirement question gets its own section in the contract guide.

6. How does it handle hostile input?

If the system reads email, documents, web pages or customer messages, someone can write instructions into them. The OWASP Gen AI Security Project, in its 2025 Top 10 for LLM applications, ranks prompt injection first and says foolproof prevention is unclear because of how these models work.

So do not ask whether the agency is “secure”. Ask what the agent is allowed to do if it is fooled. OWASP’s own guidance is least-privilege access, with the application holding the credentials rather than the model, and human approval for privileged actions. A good answer names what the agent can read, what it can write, and which actions wait for a person.

7. What do I receive if we part ways?

Ask for the handover package as a list: repository, prompts, evaluation set, configuration, documentation, and the accounts everything runs on. Then ask whether those accounts are yours from day one. Paying for code does not automatically make you its owner, and the full set of layers is in who owns the code when an AI agency builds your agents.

The test is simple: if the agency vanished tomorrow, could a different builder run and change the system from what you hold?

8. What would make you say no?

A good builder turns work down. Ask what it would refuse to build, and what it would tell a client to do first. Agencies that say yes to everything are usually pricing the unknowns into hours.

The same goes for the refund. The FTC’s complaint was partly about guarantees that sounded generous and were rarely paid. Ask for the refund terms in the contract, with the conditions, rather than on the sales page.

The eight questions side by side

Question	A strong answer	A warning answer
Still running	Named live system, owner you can contact, failure history	Screenshots, logos, “confidential”
Human involvement	A defined metric with a denominator	A single percentage, no definition
Done	Written test on your inputs, pass rate stated	”We iterate until you’re happy”
Who does the work	Named people, subcontractors bound by your terms	”Our team”
Running cost	Monthly estimate, your API key	Usage hidden inside an invoice
Hostile input	Listed permissions and approval steps	”We use the latest security”
Leaving	Itemized handover on your accounts	Repo stays with the agency
Saying no	Specific things they decline	Everything is possible

How we answer them

The prices are public so you can test us against the list. A workforce sprint starts from $9,500, fixed before we start, with a written scope. You run the accounts and the API keys, and change is a new written order, not a surprise invoice. We explain why that works in how fixed-price AI development stays fixed.

If you cannot yet answer question three because you do not know what to build, that is the AI operations audit: $1,900, fixed, 10 working days, with a full refund if we find nothing worth building.

Start async

Send us the proposal you are weighing, or describe the process you want built, at /start. You get a written reply within one business day, no meetings, and we will answer the eight questions above in writing for our own work.