Eight criteria for choosing an AI vendor — and how to run a proof of concept
Most AI proofs of concept end with everyone agreeing it was interesting and nobody able to say whether to buy. That is a design problem, not a tooling problem. Here are the criteria worth weighting and the PoC structure that forces a decision.
AI Tools Hub Editorial · published · updated
The eight criteria
| Criterion | Weight | What to actually check |
|---|---|---|
| Fit to the specific task | High | Test on your work, not the vendor's demo |
| Contractual training exclusion | High | In the DPA, not the marketing page |
| Certification | High | Current SOC 2 Type II report, under NDA |
| Identity integration | High | SSO and SCIM against your provider |
| Audit logs | High | Exportable to your SIEM |
| Contracting and billing | Medium | Entity, currency, invoice terms |
| Support | Medium | Language, hours, escalation path |
| Vendor viability | Medium | Funding, customers, whether it exists in three years |
A PoC that produces a decision
- One task, chosen because it is frequent and measurable
- Twenty to thirty participants, from the team that actually does that task
- Four weeks, with a decision date set before it starts
- Written success criteria — what number, measured how, at what threshold
- A named owner who reports the result, whatever it says
The criteria have to be written before the pilot begins. Agreeing them afterwards means agreeing with whatever happened, which is how organisations end up renewing tools nobody uses.
The questions that end evaluations early
- Is training exclusion in the contract? If not, and you handle customer data, stop here
- Is there SCIM? If not, your access review will find leavers with accounts
- Can we get an invoice in our currency? Card-only blocks many corporate buyers
- What happens to our data at termination? A vague answer is an answer
How long should a PoC run?
Four weeks. Shorter and people are still learning the tool; longer and it becomes an unmanaged deployment that nobody decides about.
Should we pilot more than one vendor at once?
Two at most, with the same task and the same criteria. Three fragments attention and you end up comparing impressions.
What if the result is ambiguous?
That is a result — it means the gain is not large enough to be obvious, which usually means not now. Record the reason so the next evaluation starts further along.
Related
More articles
Opinion
Why AI rollouts fail in Japanese companies — four structural reasons
For business
Enterprise AI rollout checklist — SOC 2, ISO 27001, DPAs and personal data
For business
Japanese-developed LLMs for business — ELYZA, Sakana AI, NTT tsuzumi and PLaMo
Comparison
AI coding tools compared — Cursor, Copilot, Claude Code, Windsurf and Codeium