Before you sign with any AI agent vendor, make them answer 30 specific questions across eight areas: identity and security, data rights, evaluation evidence, observability, human oversight, model change and incidents, compliance evidence, and exit. For each one, you should know in advance what a good answer looks like and which answer ends the conversation. The checklist below gives you both, plus a scoring method you can drop straight into an RFP. Ask for documents and demonstrations, not reassurance.
This checklist assumes you have already made two earlier decisions. First, that buying (or a hybrid) is right for this workflow; if you have not settled that, start with our build vs. buy decision framework for AI agents. Second, that the product is genuinely an agent. Run the nine-question agent-washing rubric first to confirm the vendor is not selling a relabelled chatbot. This checklist covers what comes after: whether a real agent is safe, provable, and leavable.
It also deliberately skips pricing mechanics. Fee structures, per-task normalization, and integration cost estimates are covered in our guide to the AELA pricing pivot and agent contract structure, and outcome-linked fees are covered in outcome-based agent pricing.
Why do AI agent RFPs need different questions?
AI agents need different RFP questions because they act, not just answer. An agent holds credentials, calls tools, writes to business systems, and changes behavior when its underlying model changes. Standard SaaS questionnaires check uptime and encryption; they rarely ask who the agent acts as, whether you can replay its decisions, or what happens when its model is retired.
The evidence that this gap matters is piling up, though most of it comes from vendor-run surveys, so read it as directional. Salt Security’s State of Agentic AI and API Security 2H 2026, a vendor survey of more than 380 organizations, found 49.8% had confirmed or suspected an unintended or unauthorized agent action in the past year, and only 12.5% could consistently trace an agent’s full activity. A Cloud Security Alliance study commissioned by Zenity of 445 respondents found 53% had seen agents exceed their intended permissions, 47% had an agent-related security incident in the past year, and only 8% said their agents never exceed permissions.
The commercial stakes are just as clear. Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Every one of those three causes is something a sharper RFP can surface before you commit.
How should you use this checklist?
Send all 30 questions in writing, require written answers with supporting documents, and score every response against criteria you fixed before reading any of them. Mark a handful of questions as must-haves that disqualify a vendor outright. Then use a live session to probe the weakest answers, and verify the strongest claims in a paid pilot.
Three habits make the difference between an RFP and a brochure exchange:
- Ask for artifacts. A vendor who can only describe a control in prose may not have it.
- Ask for the contract clause. If the answer matters, it belongs in the agreement, not the sales deck.
- Ask engineering too. Inconsistency between sales and engineering answers is itself a finding.
A. Identity, permissions & security
The first group decides how much damage an agent can do on its worst day. An agent’s permissions define its blast radius, so start here and treat weak answers as disqualifying. Our analysis of the agent privilege crisis explains why shared, over-scoped credentials are the most common root cause of agent incidents.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 1 | What identity does the agent act under, and how are its credentials scoped and rotated? | Its own non-human identity, least-privilege scopes per tool, short-lived credentials with automated rotation | A shared admin or service account |
| 2 | Can we restrict actions per tool and data source, and require approval for write or delete actions? | A configurable policy layer you control, down to individual operations | Permissions are fixed by the vendor |
| 3 | How do you defend against prompt injection and agent hijacking, and when were you last red-teamed? | Layered defenses plus dated third-party red-team results with remediation status | “Our model is safe” |
| 4 | How is our data isolated from other tenants? | Tenant isolation at storage, retrieval, and memory layers, shown in an architecture diagram | A shared index or vector store across customers |
| 5 | Which MCP servers and third-party tools does the agent call, and who vets them? | An allowlist, a maintained inventory, and a named vetting process | Nobody can list the tool supply chain |
Question 3 deserves extra weight. In a large-scale red-teaming competition run by NIST’s CAISI with Gray Swan and the UK AI Security Institute, more than 400 participants made over 250,000 attack attempts against 13 frontier models, and at least one hijack succeeded against every target model. No vendor can credibly claim immunity. The good answer is evidence of testing, layered controls, and limited permissions that contain a successful attack.
B. Data rights
Data-rights questions decide whether your prompts, records, and workflows become someone else’s asset. The answers must be contractual, and they must flow down to every model provider and subprocessor the vendor uses. “We don’t do that” from a salesperson is worth nothing if the upstream model provider’s terms say otherwise.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 6 | Do you or your model providers train on our inputs or outputs? | Contractually never, with the same commitment flowed down to subprocessors | Opt-out only, or “only anonymized data” |
| 7 | What is stored (prompts, tool outputs, memory), where, and for how long? | Itemized storage map with configurable retention per data type | Indefinite logs, or “we keep it to improve the service” |
| 8 | Who owns outputs, fine-tuned artifacts, prompts, and workflow configurations? | You do, stated explicitly in the contract | The vendor owns “derived” assets or configurations |
| 9 | Which subprocessors and model providers see our data, and in which regions? | A named list with regions and advance notice of changes | “It varies” |
US federal buyers already have a template here. OMB memo M-25-22, issued April 3, 2025, binds federal agencies only, but it is a useful template for private buyers: it directs agencies to bar vendors from training on non-public agency data without consent, and it emphasizes knowledge transfer, data and model portability, clear licensing, and pricing transparency.
C. Evaluation evidence
Evaluation questions separate vendors who measure their agent from vendors who demo it. You want task-level accuracy on work that resembles yours, a documented method, honest failure examples, and an automated gate that stops regressions from shipping. A single headline accuracy number with no methodology is marketing, not evidence.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 10 | Show task-level accuracy on workloads like ours, with methodology. | Eval set description, confidence intervals, and real failure examples | Demo-only evidence or a single headline number |
| 11 | Can we run our own eval set in a paid pilot before signing? | Yes, with written exit criteria agreed before the pilot starts | The “pilot” is a sales-led demo on curated data |
| 12 | How do you measure hallucination and wrong-action rates in production? | Defined metrics, sampling or automated checks, and reporting you can see | “Customers tell us if something is wrong” |
| 13 | What regression testing runs before any prompt, tool, or model change ships? | An automated eval gate that blocks releases below threshold | Manual spot-checks |
Question 11 is the most practical lever in the RFP: fix the eval set, threshold, and pilot length in writing before it begins.
D. Observability & audit
Observability questions decide whether you can reconstruct what the agent did, prove it to an auditor, and catch problems before the invoice does. Remember the Salt Security finding: only 12.5% of surveyed organizations could consistently trace an agent’s full activity. Make full, exportable traces a must-have.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 14 | Do you provide a full per-run trace of inputs, reasoning steps, tool calls, and outputs? | Yes, per run, with model and policy version attached | Summary logs or final outputs only |
| 15 | Can we export logs to our SIEM, and what is the retention? | Streaming export in a documented format, retention you configure | A vendor-only dashboard |
| 16 | Can you replay a past run and explain why the agent acted as it did? | Replay with the same inputs, versions, and tool responses | “The model is non-deterministic, so no” |
| 17 | Do you alert on cost spikes, loops, and anomalous behavior? | Configurable thresholds with alerts to your channels | You find out from the invoice |
Traces are also your incident evidence. If the vendor cannot hand you a complete run record, the first hours of an AI agent incident response become guesswork.
E. Human oversight
Oversight questions decide where people stay in the loop and how fast they can take control back. The right design is risk-tiered: routine, reversible actions run autonomously, while irreversible, high-value, or externally visible actions wait for approval. All-or-nothing autonomy is a design flaw, not a feature.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 18 | Which actions require human approval, and can we configure that? | Risk-tiered approvals you set per action type and value | All-or-nothing autonomy |
| 19 | What triggers escalation to a human, and what context is handed over? | Defined triggers, with the full case, trace, and agent rationale passed to the reviewer | Escalation drops a bare ticket with no context |
| 20 | Is there a kill switch, how fast does it work, and can we use it ourselves? | A customer-operated stop that halts actions in seconds and blocks automatic restarts | “Open a support ticket” |
| 21 | What does the agent do when it is uncertain or a tool fails? | Fails safe, stops the action, and flags it for review | Improvises or retries indefinitely |
F. Model change, incidents & SLAs
This group covers the risk most buyers miss: the product you evaluate is not the product you will run in six months. Models get upgraded, swapped, and retired upstream. You need contractual notice, the right to re-validate or roll back, and incident terms measured in hours rather than adjectives.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 22 | Which models power the agent, and how much notice do we get before a swap or upgrade? | Named models, a contractual notice period, and an option to pin a version | Silent model changes |
| 23 | Can we require re-validation, or a rollback, if a new version degrades quality? | Yes: re-run of your eval set before cutover and a contractual rollback right | “All customers move together” |
| 24 | What are the deprecation terms if an upstream model is retired? | A migration path at no cost to you, with re-validation included | “That’s the model provider’s problem” |
| 25 | What are your incident notification times, severity tiers, and service credits? | Notification in hours by severity tier, a postmortem commitment, credits that bite | An uptime-only SLA, or notice “promptly” |
OMB M-25-22 again offers a template: it encourages contract terms requiring vendors to monitor performance and roll back a new version that fails agreed standards.
G. Compliance evidence
Compliance questions should be answered with certificates, scopes, and mappings, not adjectives. “Aligned with” and “working toward” are the two phrases to listen for: they usually mean no audit has happened. Confirm that the audit scope actually includes the agent platform, not just the vendor’s billing system.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 26 | Do you hold a current SOC 2 Type II report whose scope covers the agent platform? | Yes, with the AI components explicitly in scope | Type I only, or AI components out of scope |
| 27 | Are you certified to ISO/IEC 42001, or do you have a dated roadmap with a named auditor? | Certificate, or a dated plan with the certification body named | “Aligned with ISO 42001” |
| 28 | How do you meet EU AI Act Article 50 transparency and GPAI obligations, and how do you classify our use case for high-risk purposes? | A written mapping with a named owner that reflects the current timelines | No mapping, or confusion about which dates apply |
On question 28, timing matters. Article 50 transparency and general-purpose AI obligations have applied since August 2, 2026, while high-risk obligations were pushed to December 2027 and August 2028 by the Digital Omnibus. A good vendor knows which obligations are live and which are delayed, and does not treat the delay as permission to do nothing; our breakdown of the EU AI Act delay trap explains why. Useful reference frameworks for scoring this group include NIST AI RMF 1.0, ISO/IEC 42001:2023, and the EU Model Contractual Clauses for AI (MCC-AI) published in March 2025. The NIST AI Agent Standards Initiative, launched February 17, 2026, adds agent security and identity to that picture; ask vendors how they are tracking it.
H. Exit & proof
The last group tests whether you can leave and whether anyone has actually stayed. Lock-in worry is widespread: in a Zapier survey of 500 US enterprise executives, 81% were at least somewhat concerned about AI vendor dependency, and of the 66% who had tried migrating between AI platforms, only 42% said it went smoothly.
| # | Question | Good answer | Red flag |
|---|---|---|---|
| 29 | What can we export on exit (data, prompts, configs, evals, memory, traces), in what format, on what timeline, and do we get a deletion certificate? | Everything listed, in documented open formats, within a fixed window, followed by certified deletion | Data only, in a proprietary format, “on request” |
| 30 | Give us three comparable reference customers in production for six months or more, and tell us what share of your pilots reach production. | Named, callable references in your size and use case, and a straight answer on pilot conversion | Logos only, pilot-stage references, or a refusal to share conversion |
The model is rarely what traps you. A Stack Overflow survey of 520 technical leaders found only 8% use a single model provider, and the hardest things to migrate were integrations to internal systems and MCP servers (53%) and multi-step workflows (52%). That is why question 29 asks for configurations, workflows, evals, and traces, not just your data.
How do you score AI agent vendor responses?
Score vendors in two passes. First, apply pass-fail must-haves: any vendor with a disqualifying answer is out, whatever the price. Second, score every remaining answer zero, one, or two against your written criteria, multiply by category weights set to your risk profile, and compare totals. Then verify the top scorer in a paid pilot.
A workable scoring model:
- Pick your must-haves. For most agents with write access, a sensible starting set is questions 1 (own identity), 6 (no training on your data), 14 (full trace), 20 (customer-operated kill switch), 22 (notice before model changes), and 29 (full export). A red-flag answer on any of these disqualifies.
- Score each answer 0, 1, or 2. Zero is the red flag or no answer, one is partial or prose-only, two is the good answer backed by an artifact or contract clause.
- Weight the categories by risk. A customer-facing agent that can issue refunds should weight groups A, D, and E most heavily. An internal research assistant with read-only access can shift weight toward C (evaluation) and H (exit). Fix the weights before you read any responses.
- Discount unverified claims. Anything scored two in writing but not demonstrated in the live session or pilot drops to one.
- Record the reasoning. A one-line justification per score makes the decision defensible to finance, legal, and the board.
If you want an outside reference for structuring the process, the Open Contracting Partnership’s Buying AI key questions and the GSA AI acquisition resource guide are both written for public buyers but translate well to mid-market procurement.
The bottom line
A good AI agent vendor will welcome these 30 questions, because each one has an answer they have already documented. A weak vendor will push back on the questions themselves, offer a demo instead of a trace, or promise that the contract can be “worked out later.” That reaction is the most useful data point the RFP produces.
We help mid-market teams write agent RFPs, score vendor responses, and design the pilot that proves the winner, and we build custom agents when buying is the wrong answer. If you are about to send an RFP or are staring at three proposals that all sound the same, talk to us. Bring the shortlist and the workflow.
Sources
- Salt Security State of Agentic AI and API Security 2H 2026, via Campus Technology
- Cloud Security Alliance: AI agent scope violations study, commissioned by Zenity
- NIST CAISI: Insights from a large-scale AI agent red-teaming competition
- OMB M-25-22: Driving Efficient Acquisition of Artificial Intelligence in Government
- NIST: Announcing the AI Agent Standards Initiative
- Zapier: AI vendor lock-in survey
- Stack Overflow: Survey says you’ll swap the model, but everything you build around it is staying put
- Open Contracting Partnership: Buying AI, key questions to ask
- GSA: Buy AI resource guide