Trusting AI Requires More Than Trusting the Vendor
Satya Nadella recently posted about how to run powerful AI inside the enterprise, and one line in it should change how risk teams think. Validation, he wrote, must be independent of the thing being validated. He also said a model provider’s assurances don’t relieve the customer of responsibility. His post is about enterprises containing their own AI, but the principle applies to anyone relying on a third party.
AI is creating a challenge the assurance industry has never had to solve at scale. Traditional software typically behaves within predefined boundaries. Modern AI systems can reason, invoke tools, access enterprise data, and act on behalf of users.
The AI supply chain is rapidly becoming more complex than the software supply chain. As organizations increasingly rely on third-party AI, they are accepting risk from systems they did not build, cannot inspect, and often cannot independently test. That changes the question from "Does this vendor have AI?" to "What evidence exists that this AI operates within verified security boundaries?" The future of AI trust will be determined by how that question gets answered.
If a vendor is the only source of evidence about its own AI controls, you don’t have assurance. You have a claim. The vendor built the system, wrote the policies, and is now describing how well both work. Nadella’s point is that this arrangement shouldn’t be accepted, and he calls for industry standards where existing ones fall short.
What Programs Are Getting Today
In practice it comes down to two things.
The first is a questionnaire. A few AI questions get added to the existing set, the vendor answers them, and the answers go in the file. Nobody can say whether those were the right questions, whether the answers are accurate, or whether two vendors who both said “yes” meant the same thing. This is self-reporting, and no one independent has looked.
The second is a governance assurance, such as ISO 42001. That’s a real step up, and it’s independent. It tells you the vendor has a management system for AI: policies, roles, risk processes, accountability.
But governance and security are different questions. A management system standard tells you the vendor has a process for managing AI risk. It doesn’t tell you whether the model’s access to your data is limited to what it needs, whether prompt injection has been tested, whether actions are logged in a way that survives an investigation, or whether anyone can shut an agent down mid-task. A vendor can have mature AI governance and still have an agent with far too much access.
Where the Risk Actually Lives
Nadella’s post puts the weight on technical controls: separating the model from the harness that orchestrates it, limiting the action space, keeping safeguards outside the model, logging every meaningful action, and being able to contain a model from the start. The relevant questions are security questions:
- What can the AI reach, and who approved it?
- What can it do without a human signing off?
- Are those limits enforced outside the model, or does the system depend on the model behaving?
- Is there a tamper-resistant record of its actions?
- Can an authorized person stop it, and how fast?
- Has anyone tested it against attack, including misuse of connected tools?
- Which model providers and tools sit underneath, and what happens if one is compromised?
- When something fails, what does the customer get told, and when?
Questionnaires can ask about these but can’t verify them. Governance assurances generally don’t test them.
What TPRM Programs Need
Independent assurance over AI security controls, with a few properties:
- Independent. The party testing the controls didn’t build them and has no stake in the result.
- Explicit scope. You can see which AI systems were examined and which weren’t.
- Operating effectiveness. Controls were tested in practice, not just documented.
- Current. Models, prompts, and tool permissions change fast, so stale evidence misleads.
- Comparable. A common set of criteria lets you evaluate many vendors without a custom review for each.
Governance assurance and security assurance answer different questions, and a mature program should want both. The mistake is treating the first as a substitute for the second.
What to do This Quarter
Pull your last few AI-related vendor assessments and sort the evidence into three piles: self-reported answers, governance assurance, and independently tested security controls. Most programs will find the third pile nearly empty. That’s your gap.
Then change what you ask for. Skip the longer questionnaire. Ask which AI systems are in scope, who tested the security controls, against what criteria, and how recently.
The Caveat
Security assurance doesn’t prove a model’s outputs are correct or that it’s aligned with anyone’s intent. It tells you the vendor has limited what the AI can reach and do, can see what it’s doing, and can contain a failure. That shrinks the blast radius without eliminating the risk, and a good program should say so plainly.
But it’s the part buyers can demand today, and the part most programs are missing.