What You Need to Know

Today’s AI systems depend on interconnected models, agents, data sources, tools, cloud platforms, and infrastructure providers. A weakness anywhere in that ecosystem can affect the security and reliability of the resulting AI service.

CISOs and AI vendor security leaders should keep these points in mind:

• The threat surface has moved beyond the model

• Agentic AI moves risks into real-world actions

• Untrusted content is a new attack vector

• Identity and accountability break down with autonomous agents

• Third- and nth-party dependencies obscure accountability, so independent assurance matters

Introduction

The AI threat landscape is shifting from attacks against an individual model to attacks against an entire AI operating environment.

The model is only one part of that system. AI systems may also include retrieval databases, persistent memory, autonomous agents, identity services, APIs, external tools, cloud infrastructure, and human approval workflows. AI systems may appear simple, but behind the scenes they are quite complex.

According to the World Economic Forum Global Cybersecurity Outlook 2026, the top three cybersecurity issues related to generative AI from respondents are data leaks, advancement of adversarial capabilities, and technical security of the AI systems themselves. This reflects the growing awareness by organizations of the need for assessing AI security. For security leaders, the central question is therefore no longer simply, “Is the model secure?” It is:

Can I prove that the model, the system in which it operates, and the third parties supporting it are secure?

Emerging Threats to AI Systems

The AI threat landscape is evolving rapidly, making it hard for organizations buying AI technology and AI system vendors to keep pace. These are some of the emerging threats to AI systems we have identified that need to be considered.

1. Training-data, model, retrieval, and memory poisoning through third parties

AI models depend on large collections of training data, fine-tuning data, evaluation sets, retrieved information, and in some cases persistent memory. An attacker who corrupts any of these inputs may be able to influence how the model behaves.

What’s new is that poisoning can enter through third-party datasets, open-source models, fine-tuning services, retrieval systems, knowledge bases, persistent memory, or other components added later in the lifecycle. This requires continuous evaluation of data integrity and provenance across the full AI system, not just the original model.

In retrieval-augmented generation systems, poisoned content can live inside a knowledge base and surface only when a particular query is asked. Because the malicious content lives in the knowledge base rather than the prompt, it can persist across many sessions and users, quietly steering answers, corrupting decisions, or serving as a delivery mechanism for indirect prompt injection.

Memory creates a similar risk. Some agents retain user preferences, prior decisions, or information learned during previous interactions. An attacker may attempt to insert false facts or malicious instructions into memory, causing the agent to reuse poisoned information in future sessions after the original attack has disappeared from view.

The risk is amplified when ingestion pipelines pull from open or loosely governed sources, or when agents can write to memory, retrieval indexes, shared repositories, or other systems that later become trusted context. A single poisoned document, web page, code repository, or email may quietly redirect workflows, leak information repeatedly, or spread corrupted content to other agents and knowledge repositories.

2. Automation of jailbreaks and safety-control circumvention

Jailbreak methods are becoming more automated and sophisticated. They may use long, multi-step conversations, encoded instructions, role-playing scenarios, adversarial suffixes, or instructions hidden in images and documents. Highly capable models may successfully interpret sophisticated or obfuscated attack inputs that less capable models fail to understand, potentially increasing exposure to advanced jailbreak techniques.

The risk becomes significantly greater when a jailbroken model has access to sensitive information, code execution, cloud consoles, communication systems, or physical operations. In those circumstances, safety circumvention can become a security incident rather than a content-moderation failure.

3. Indirect prompt injection

Indirect prompt injection occurs when malicious instructions are embedded in content that an AI agent reads, such as an email, webpage, document, image, calendar invitation, or code repository.

The agent may mistake those instructions for legitimate directions and act on them. For example, manipulated content could attempt to make an agent retrieve sensitive information, alter a workflow, or send data to an external destination. While there is not a large body of public examples, some examples highlight the dangers. For example, a plaintiff in a court case included concealed prompts in court filing documents, instructing any AI model to agree with his position and treat the clerk's previous ruling against him as an error. The most dangerous configuration combines three capabilities:

1. Access to private data
2. Exposure to untrusted content
3. The ability to communicate or act externally

When all three are present, a single poisoned source may provide an attack path from initial manipulation to data exfiltration or unauthorized system activity.

4. Agent identity and excessive agency

Traditional identity and access management (IAM) was built for two kinds of actors: human users and relatively predictable non-human service accounts. Agentic AI does not fit neatly into either category. Within a single workflow, an agent may act on behalf of a specific person in one moment, operate as an automated process, call external tools, and even spawn sub-agents to complete a task.

This creates identity and accountability challenges that conventional IAM controls were not designed to address. When an agent takes an action, it may be difficult to answer questions like:

• Who was the agent acting for?
• What authority was it actually granted, and for what purpose?
• Did it delegate work to another agent or tool, and did that authority expand along the way?
• If something goes wrong, which component in the chain was responsible?

These questions become harder still when agents share broad, long-lived credentials, inherit permissions implicitly, or create sub-agents without clear limits or expiration.

An agent may struggle to distinguish between what a user explicitly requested, what an external document suggested, what another agent delegated, and what a tool claimed was necessary. It may have legitimate authority but use that authority in response to an untrusted or manipulated input. The risk increases when agents use broad, persistent credentials or can create sub-agents without clear limits, making incident investigations particularly challenging.

5. Multi-agent coordination and cascading failures

Organizations are increasingly deploying AI systems composed of multiple AI agents that collaborate, delegate work, and share information. While this approach can improve scalability and specialization, it also introduces new security risks that don’t exist in single-agent architectures.

An attacker who compromises, manipulates, or poisons one agent may be able to influence other agents that trust its outputs. Malicious instructions, false information, or unauthorized actions can propagate through delegation chains, creating cascading failures that become difficult to detect and investigate. Multi-agent environments may also introduce risks such as agent-to-agent prompt injection, unauthorized delegation, agent impersonation, recursive task loops, and excessive resource consumption.

6. Compromised tools, connectors, and agent platforms

Every tool or connector available to an AI system is both a software dependency and a potential source of instructions entering the model’s context. A compromised connector, plugin, API, or MCP server could steal credentials, return manipulated information, inject malicious instructions, or cause an agent to invoke additional compromised services. The resulting risk depends on what the component can access, whether it can modify data or systems, and whether it can communicate externally.

7. Model, dataset, and software supply-chain attacks

A provider that appears to deliver a single AI product may depend on a complex chain of upstream and downstream organizations for third-party models, datasets, embeddings, container images, open-source packages, development frameworks, and hosted services. This leads to a range of threat vectors that include poisoned datasets, backdoored model weights, malicious model files, compromised model repositories, vulnerable inference servers, unsafe agent templates, and malicious software packages.

These nth-party relationships create significant challenges for security leaders. An organization may have contractual and risk-management visibility into its AI system vendor, but little insight into the model provider, hosting environment, data source, or tool developer supporting the vendor’s system.

8. Attacks against compute, orchestration, and cloud environments

AI workloads concentrate valuable data, intellectual property, credentials, and expensive computing resources in shared infrastructure. AI activities happen across a broad range of infrastructure and applications. Attackers may target cloud management interfaces, container environments, workload schedulers, GPU drivers, notebooks, storage systems, or secrets embedded in jobs and container images. Model checkpoints created during training can also become targets because they may contain valuable versions of a model’s capabilities.

AI infrastructure additionally creates opportunities for economic denial-of-service attacks. An attacker may deliberately trigger long reasoning operations, recursive agent loops, repeated tool calls, and GPU-memory exhaustion. A relatively inexpensive request can generate substantial costs across inference services, databases, and external APIs.

A recent example comes is the attack on Hugging Face. They reported that an intrusion into production infrastructure was conducted through an autonomous agent framework, which turned out to be OpenAI testing frontier models. Hugging Face’s data pipeline was abused to run code on a processing worker. The agent then escalated privileges, collected more credentials, and moved to other systems, i.e., classic threat actor tradecraft. The incident highlights several risks associated with AI: the pace of frontier model development and an agent's ability to identify and abuse vulnerabilities and build complex attack chains, and the need for adequate guardrails for agentic systems.

Recommendations for Third-Party Risk Management (TPRM) Teams

The uncomfortable reality is that many companies are adopting AI faster than they're building AI threat and risk expertise. As a result, expecting TPRM to independently assess AI security at a deep technical level is often unrealistic. So how do you get your TPRM program up to AI speed?

1. Establish an AI Security-specific Process
Determine how you are going to assess the security of the AI system that can be done in an efficient, repeatable, and comparable process. The challenge is current assurance instruments aren't fit for purpose because they lack scalability, relevance, and reliability. Using these instruments can introduce more delays and friction into the assessment and procurement process because they can't answer the question of how secure the AI system really is with confidence.

2. Assess the complete AI system
Determine how you are going to assess the security of the AI system that can be done in an efficient, repeatable, and comparable process. The challenge is current assurance instruments aren't fit for purpose because they lack scalability, relevance, and reliability. Using these instruments can introduce more delays and friction into the assessment and procurement process because they can't answer the question of how secure the AI system really is with confidence.

3. Require independent assurance
TPRM functions should establish assurance requirements based on the sensitivity of the data, the autonomy of the system, and the potential impact of its actions.

Recommendations for AI System Vendors

These recommendations apply to the broad ecosystem of model developers, model hosts, cloud providers, AI application vendors, agent-platform developers, connector providers, and data suppliers, all of which could be part of an AI system.

1. Understand and document the AI system
Maintain a clear understanding of how the AI system works, including the organizations, technologies, data sources, and services that support it.

2. Ensure transparency and traceability
Be able to identify where important system components, data, and outputs come from, and how they have changed over time.

3. Limit access and authority
Ensure AI systems have only the access and permissions needed to perform their intended functions, with appropriate oversight for significant actions.

4. Protect information used by the AI system
Safeguard the information, knowledge sources, and records that AI systems rely on to make decisions and take actions.

5. Monitor AI system behavior
Maintain visibility into how AI systems operate, what information they use, the actions they take, and any unusual or unexpected behavior.

6. Evaluate the security of the complete system
Assess the security and resilience of the entire AI system, including people, processes, technologies, data sources, and external dependencies.

3. Provide independent, validated assurance
Support prospects and customer trust through independent assessments, certifications, or other forms of validated assurance.  

Conclusion

AI is rapidly becoming one of the largest sources of third-party risk in the enterprise. Unlike traditional software, AI systems rely on interconnected models, agents, external data sources, retrieval systems, cloud infrastructure, and numerous third- and nth-party providers, making accountability and security more difficult to assess. As a result, vendor questionnaires and legacy assessment approaches are often insufficient to evaluate the true risk of an AI-enabled system. TPRM teams that establish AI-specific assessment requirements and demand independent, validated assurance like HITRUST AI Security will be better positioned to identify hidden risks, accelerate procurement decisions, and protect their organizations from emerging threats that can originate anywhere in the AI supply chain.

Vendors standardizing on HITRUST AI Security can gain a competitive advantage that provides transparency to buyers and enables them to remove time and friction in the procurement process. This makes it easier for buyers to perform their evaluation efficiently and effectively to meet the needs at the speed of their organization.

Take the next step

Reach out today to learn how the HITRUST can help your organization evaluate AI security, validate security controls for current and emerging AI threats, and build trust across ecosystems.