AI Vendor Assessment Questionnaire: 25 Questions to Ask Before You Buy

Quick answer: Before adopting any AI tool, ask vendors 25 questions across five areas: data handling (where it goes, whether it trains models), security and access, model transparency (provenance, known limitations), regulatory posture (their EU AI Act role and yours), and commercial terms (liability, exit, change notice). Most of your AI risk is vendor risk — you inherit their data practices, their model behavior, and their compliance gaps the moment you sign.

When an AI tool leaks your client data or quietly ships an emotion-recognition feature that's banned in the EU, "the vendor did it" won't help you — as the deployer, much of the exposure is yours. Here are the questions that surface problems before the contract does.

Why AI Vendors Need Different Questions Than Regular SaaS

• Training leakage. Regular SaaS stores your data; AI SaaS may learn from it and surface fragments to other customers.
• Silent behavior change. Vendors swap underlying models without notice; outputs change, your validated workflows break.
• Inherited compliance. Under the EU AI Act, your obligations depend on what the system does and what tier it falls into — a vendor feature update can move you between tiers.
• Fourth parties. Many "AI vendors" are thin wrappers over another provider's model. Your data's real destination is one hop further than the contract suggests.

The 25-Question Assessment

Section A: Data Handling (the dealbreakers)

1. What customer data does the system collect, process, and store — and in which jurisdictions?
2. Is our data used to train or fine-tune your models or any third party's models? Is opt-out contractual, account-level, and default-on for our tier?
3. How long are prompts/inputs and outputs retained? Can we set retention to zero or near-zero?
4. Will you sign a DPA? Are you acting as processor or controller for our data?
5. Which subprocessors and underlying model providers receive our data?
6. How is our data deleted at termination, and how do you evidence deletion?

Red flags: training opt-out only available on a higher tier; vague "we may use data to improve services" language; refusal to name the underlying model provider.

Section B: Security and Access

1. Which certifications do you hold (SOC 2 Type II, ISO 27001)? Share the report under NDA.
2. Is our data segregated from other tenants'? How?
3. Do your staff or contractors review customer prompts/outputs? Under what controls?
4. What's your breach notification commitment in hours, contractually?
5. Do you support SSO, role-based access, and audit logs at our tier?

Section C: Model Transparency and Performance

1. What model(s) power the product? Built in-house, fine-tuned, or wrapped via API?
2. What are the documented limitations, failure modes, and known bias issues?
3. How are we notified before the underlying model or its behavior changes materially?
4. What accuracy/quality benchmarks do you publish, and how are they measured?
5. Can outputs be traced or logged on our side for audit (export of usage logs)?

Section D: Regulatory Posture (EU AI Act and beyond)

1. What is your role under the EU AI Act for this product — and what do you understand ours to be?
2. Which risk tier do you classify this system into, and on what reasoning? (The four tiers, explained.)
3. Does any feature involve emotion recognition, biometric categorization, or social scoring? (These are Article 5 territory — banned in the EU since 2 February 2025.)
4. For chat or content-generation features: how do you support our Article 50 transparency duties from 2 August 2026?
5. If the system could be high-risk (e.g., HR screening, credit decisions): what provider documentation and instructions for use will you supply for the 2 December 2027 Annex III deadline? Will you provide Annex IV-style technical documentation?

Make this repeatable: the AI Governance Toolkit Pro ($99) includes this vendor assessment as a ready-to-send questionnaire, plus the risk register where the answers land, a governance charter, and a staff training deck.

Section E: Commercial Terms and Exit

1. What liability do you accept for AI output errors, data leakage, and IP infringement in generated content?
2. Who owns the outputs we generate? Any restrictions on commercial use?
3. What's the change-notice period for pricing, features, and terms?
4. What's our data export path at exit — formats, timelines, costs?

Scoring the Answers

Two hard rules: any "yes" on training without opt-out = fail for confidential data use, and any Article 5-adjacent feature = fail for EU deployment. File every completed assessment — under the EU AI Act and in enterprise procurement reviews alike, documented vendor due diligence is evidence. Results feed into your AI risk register and broader governance framework.

FAQ

What should an AI vendor assessment cover?

Five areas: data handling (especially whether your data trains models), security and access controls, model transparency and change management, regulatory posture under the EU AI Act, and commercial terms including liability and exit.

What's the single most important question to ask an AI vendor?

"Is our data used to train your or any third party's models, and is the opt-out contractual and default for our tier?"

How is AI vendor risk different from normal SaaS risk?

Add four things: your data potentially training shared models, silent model behavior changes, regulatory tier inheritance under the EU AI Act, and fourth-party model providers behind the vendor you actually signed with.

Stop approving AI tools on vibes. The AI Governance Toolkit Pro ($99) packages this questionnaire with the charter, risk register, and training deck — or get the full document set in the Complete AI Compliance Stack ($199).

This article is for general information only and is not legal advice. Consult qualified counsel for your specific situation.