What to Ask Before Hiring an AI Development Shop

You've decided your business needs custom AI. Maybe you've outgrown the no-code tools. Maybe you have a workflow that's too specific for any SaaS product to handle. Maybe you just know there's a competitive advantage hiding in your data and you need someone to build it.

Whatever the reason, you're about to spend real money on a team that's going to touch critical parts of your business. The wrong choice can cost you months and a small fortune. The right choice can change the trajectory of your company.

Here's what to ask before you sign anything.

1. "Can you show me something you've shipped to production?"

This is the single most important question. Not a demo. Not a proof of concept. Not a Jupyter notebook. Something that real users interact with every day.

AI has a demo problem. It's easy to build something that looks impressive in a controlled environment. It's much harder to build something that handles edge cases, scales under load, and doesn't break when users do something unexpected.

If a team can only show you prototypes, that's a red flag. You want a team that has been through the full lifecycle — from initial build through deployment, monitoring, and iteration.

Follow-up questions:

• How long has it been in production?
• What went wrong after launch and how did you handle it?
• What does the monitoring look like?

2. "What happens when the AI is wrong?"

Every AI system makes mistakes. Every single one. The question isn't whether your system will be wrong — it's what happens when it is.

A good team will have a clear answer for this. They'll talk about confidence thresholds, human-in-the-loop fallbacks, graceful degradation, and error handling. They'll have opinions about where automation should stop and where a human should take over.

A bad team will tell you their system is 99% accurate and leave it at that.

What you want to hear:

• "We design for failure from the start"
• "Here's how we handle low-confidence outputs"
• "We build monitoring to catch drift before users notice"

What should worry you:

• "Our model is highly accurate"
• "We'll fine-tune until the errors go away"
• Anything that suggests they haven't thought about failure modes

3. "Who owns the code and the data?"

This sounds like a legal question, but it's really a business question. If you're paying someone to build a custom AI system, you need to own what comes out the other end. Full stop.

That means:

• Source code — You should have the complete codebase, not just access to a hosted service
• Model weights — If they fine-tuned a model on your data, you own those weights
• Training data — Your data stays yours, and it shouldn't be used to train models for other clients
• Infrastructure — You should be able to take the system and run it yourself or hand it to another team

Some shops build on proprietary platforms that lock you in. If you can't take your system and walk away, you don't really own it.

4. "What's your stack, and why?"

You don't need to be technical to ask this question. What you're really looking for is whether the team makes deliberate technology choices or just uses whatever they're most familiar with.

A thoughtful answer sounds like: "For this type of problem, we'd use X because of your latency requirements and data sensitivity constraints. If your needs change, we could swap to Y without rebuilding."

A lazy answer sounds like: "We use [specific framework] for everything."

The right stack depends entirely on your requirements. Cost, speed, privacy, scale — these all push toward different solutions. A team that has one answer for every problem probably isn't thinking hard enough about yours.

Bonus points if they mention:

• How they handle model versioning
• Their approach to testing AI systems (it's different from testing regular software)
• How they think about cost per inference at scale

5. "How do you scope work, and what does pricing look like?"

AI projects are notoriously hard to estimate. Anyone who gives you a fixed bid on day one is either padding heavily or doesn't understand the problem yet.

The best teams are honest about uncertainty. They'll typically propose one of these structures:

• Discovery phase + build phase — A short, fixed-cost period to validate the approach, followed by a scoped build. This is usually the healthiest model.
• Monthly retainer — Good for ongoing development and iteration. Less good for a defined deliverable.
• Fixed scope with milestones — Works if the problem is well-understood. Risky if there's any ambiguity.

Whatever the model, make sure there are clear milestones and check-in points. You should never be three months into a project with nothing to show for it.

Red flags:

• "We'll figure it out as we go" with no structure
• A massive fixed bid with no discovery phase
• No mention of what happens if the approach doesn't work

6. "How will we know if this is working?"

Before any code gets written, you should agree on what success looks like. Not in vague terms — in specific, measurable terms.

Good metrics are tied to business outcomes:

• "Reduce manual review time by 40%"
• "Handle 80% of customer inquiries without human escalation"
• "Process incoming documents in under 30 seconds with 95% accuracy"

Bad metrics are purely technical:

• "Achieve 0.92 F1 score" (meaningless to most stakeholders)
• "Use GPT-4" (that's a tool, not an outcome)

A good team will help you define these metrics during the discovery phase and build dashboards so you can track them in real time.

7. "What does handoff look like?"

Eventually the project ends. What happens then?

You want to know:

• Documentation — Will there be clear docs for how the system works, how to maintain it, and how to extend it?
• Training — Will your team know how to operate the system without the vendor?
• Support — Is there a support period after launch? What does it cost?
• Extensibility — Is the system built so your internal team (or another vendor) can add features without starting over?

The goal is independence. A good partner builds you something you can own and operate. A bad partner builds you something that only they can maintain.

8. "What do you need from us?"

This is the question most clients forget to ask, and it's one of the most important.

AI projects require active participation from the client side. You'll need to provide:

• Domain expertise — The team building your system needs to understand your business. That knowledge comes from you.
• Data access — Models are only as good as the data they're built on. Delays in data access are the number one cause of project delays.
• Feedback — Especially in the early stages, your team needs to review outputs and provide feedback to improve the system.
• Decision-making authority — Someone on your side needs to be empowered to make decisions quickly. Projects stall when every choice needs to go through three levels of approval.

The best AI projects are partnerships, not outsourcing arrangements. The more engaged you are, the better the outcome.

The Bottom Line

Hiring an AI development shop is a bet on a team's ability to understand your problem and build a reliable solution. The technology matters, but it matters less than the team's judgment, communication, and track record of shipping.

Ask hard questions. Look for honest answers. And if something feels off during the sales process, trust that instinct — it only gets harder once the project starts.

If you're evaluating teams and want a straight conversation about whether your project makes sense, we're always happy to talk. No pitch, no pressure — just an honest assessment of what it would take to build what you're imagining.