How to Evaluate an Automation Partner: 10 Questions to Ask

Why the Vendor Decision Matters More Than the Technology

Most failed automation projects do not fail because of bad technology. They fail because of misalignment between what the business needed and what the vendor delivered. A capable partner asks hard questions up front, pushes back on unclear requirements, and builds for longevity rather than a quick handoff. A poor one says yes to everything, ships something that technically works, and disappears.

The difference between these two outcomes is usually visible before you sign a contract, if you know what to look for. Here are ten questions that separate serious partners from vendors chasing project fees.

1. Do They Understand Your Industry?

Automation is not generic. A workflow that works in e-commerce does not translate directly to logistics or professional services. The edge cases, compliance requirements, and system landscapes differ significantly.

Why it matters: An agency with relevant industry experience will anticipate problems you have not thought of yet. They will know the common integration points, the data pitfalls, and the regulatory constraints before the first meeting ends.

What a good answer looks like: Specific examples from your sector, not vague claims about "cross-industry expertise." They should reference the tools your industry uses and the typical pain points without you having to explain them.

2. Can They Show Relevant Case Studies?

Portfolios full of landing pages and chatbots tell you nothing about whether a vendor can build a multi-system workflow that handles exceptions reliably.

Why it matters: Past work is the strongest predictor of future performance. According to Gartner's vendor evaluation framework, reference cases that match your project scope reduce implementation risk by up to 40%.

What a good answer looks like: Case studies with specifics: the business problem, the technical approach, measurable results, and the timeline. Bonus if they can connect you with a reference client.

3. What Happens After Delivery?

This is where most vendor relationships reveal their true nature. Building the automation is the easy part. Maintaining it when APIs change, when your business processes evolve, or when something breaks at 7 AM on a Monday is where the real value lies.

Why it matters: Automation systems are not static. They need monitoring, updates, and occasional rework. A vendor who builds and walks away is selling you a depreciating asset.

What a good answer looks like: A clear support model with defined response times, a maintenance agreement, and transparent pricing for ongoing work. They should explain how they handle monitoring and incident response.

4. Who Owns the Code and Workflows?

This question catches more businesses off guard than any other. Some agencies retain ownership of the code they build, which means you cannot modify, migrate, or maintain it without them.

Why it matters: Vendor lock-in is expensive. If you ever want to switch providers, bring development in house, or simply understand what is running in your infrastructure, you need full ownership.

What a good answer looks like: You own everything. Code, documentation, credentials, workflow configurations. The vendor should be able to hand over a complete repository with documentation that another developer could pick up. Anything less is a red flag.

5. What Is the Pricing Model?

Fixed project fee, hourly rate, monthly retainer, or outcome based pricing all create different incentive structures. Understanding the model tells you what behavior the contract encourages.

Why it matters: An hourly model incentivizes longer timelines. A fixed fee incentivizes cutting corners to stay within budget. A retainer model incentivizes ongoing dependency. None of these are inherently bad, but you need to understand the trade-offs. Harvard Business Review's research on service contracts suggests that hybrid models (fixed scope with a maintenance retainer) tend to align incentives best.

What a good answer looks like: Transparent pricing with a clear scope definition, explicit handling of scope changes, and no hidden fees for hosting, API costs, or support.

6. How Do They Handle Data Privacy and GDPR?

If your automation touches customer data, employee data, or any personal information, GDPR compliance is not optional. Many automation projects involve moving data between systems, which creates processing and transfer obligations.

Why it matters: A data breach or compliance violation resulting from an automation your vendor built is still your responsibility. The European Commission's GDPR enforcement tracker shows that SMBs are increasingly subject to enforcement actions.

What a good answer looks like: They ask you about data flows before you ask them about GDPR. They know the difference between a data processor and data controller. They have a standard Data Processing Agreement ready and can explain how data is handled at every step of the automation.

7. What Is Their Approach to Testing?

Automation that works in a demo and automation that works in production with real data, edge cases, and concurrent users are fundamentally different things.

Why it matters: Insufficient testing is the most common cause of post-launch failures. A vendor who skips testing to hit a deadline is creating technical debt that you will pay for later.

What a good answer looks like: They describe a testing process that includes unit tests, integration tests, and user acceptance testing. They test with realistic data volumes. They have a staging environment separate from production. They plan for edge cases and failure scenarios explicitly.

8. Do They Build Custom or Use Off-the-Shelf Tools?

This question is not about which approach is better (that depends on your situation, as we explore in our post on custom vs. off-the-shelf automation). It is about whether the vendor matches the right approach to the right problem.

Why it matters: A vendor who only builds custom solutions will over-engineer simple problems. A vendor who only uses no-code tools will hit a ceiling on complex workflows. The best partners have both capabilities and are honest about when each applies.

What a good answer looks like: They evaluate your specific needs and recommend the approach that fits, even if it means a smaller project. They can articulate the trade-offs clearly: cost, scalability, maintenance burden, and flexibility.

9. What Does the Handoff Look Like?

The transition from "vendor building" to "team operating" is where many automation projects stall. If your team cannot run the system independently after handoff, you have not bought automation. You have bought a dependency.

Why it matters: Your team needs to understand what was built, how to monitor it, how to handle common failure scenarios, and who to contact when something goes beyond their ability to fix.

What a good answer looks like: Documentation, training sessions, a recorded walkthrough, and a defined support window after go-live. They should provide runbooks for common issues and escalation paths for uncommon ones.

10. Can You See It Before You Commit?

A reputable partner should be willing to demonstrate their approach before you commit to a full build. This could be a proof of concept, a working prototype, or a detailed technical design with architecture diagrams.

Why it matters: Seeing a working prototype eliminates the biggest risk in any automation project: building the wrong thing. If you are already seeing signs that your business is ready for automation, a small proof of concept is the fastest way to validate the approach.

What a good answer looks like: They offer a paid discovery or prototyping phase that delivers a working demonstration of the core workflow. This phase should have its own deliverable and timeline, separate from the full build.

Using These Questions

You do not need perfect answers to all ten. But the pattern of responses tells you a lot. A vendor who answers six of these confidently and honestly about the gaps is far better than one who gives polished non-answers to all ten.

Print this list. Bring it to your next vendor meeting. The right partner will appreciate the rigor.