How to Practice Threat Modeling in the Age of AI

Before going outside, you check the weather forecast. And before deploying software, you identify and prioritize potential vulnerabilities and threats.

This is how fundamental threat modeling is to the software development lifecycle (SDLC). Just like checking for bugs and conducting quality reviews, threat modeling is an essential step in building software applications. No competent developer would ever release a product without identifying the most likely security risks they should plan for first.

Yet that’s exactly what’s happening right now. Although AI applications may rely on some of the same infrastructure, APIs, and access tools as traditional software, the way they function is fundamentally different. Instead of the known code paths, predictable logic, and stable states that define traditional software, AI applications introduce reasoning and behaviors that can’t be modeled in the same way. The result has been an explosion in AI-related vulnerabilities.

But this doesn’t mean threats can’t be modeled across AI. Instead, reducing the security risks of LLMs, AI agents, and the systems they integrate with requires a different approach to threat modeling.

How traditional threat modeling falls short

The usual approach to threat modeling goes something like this: During the design stage of an application, developers will create a diagram or model of what they are building. Using this view, they’ll then attempt to identify anything that might go wrong. This could involve manually picking out potential vulnerabilities in the software, assessing a knowledge base such as MITRE ATT&CK, or using a framework like STRIDE to help model and identify threats. 

The purpose of all of this is to help developers proactively locate risk so that they can either create safeguards or mitigate these threats altogether. But threat modeling can provide numerous other benefits as well. For example, it can help create cross-functional alignment by drawing together input from multiple stakeholders, help enforce regulatory compliance, and give incident response teams a pre-built playbook for real threats. All of this is why threat modeling is seen as a cornerstone of software development. So why is it falling so short when it comes to AI?

One reason is because of the probabilistic nature of AI. Whereas traditional software will always return the same result when given identical instructions, AI outputs will often vary. Instead of executing a command, it uses the input it’s given to calculate the most likely (or most probable) answer. Because AI systems will also weigh a variety of other factors, such as the wider context and its own training data, this can produce a level of unpredictability that makes classic threat modeling insufficient.

The nature of AI also introduces new vulnerabilities that can be difficult to model and predict. For example, the majority of AI models are designed to be helpful. If you input a command, it will do its best to provide an answer you think will be valuable. But this behavior also makes AI systems vulnerable to techniques such as prompt injection, manipulation, and coercion. Even with firm guardrails in place, bad actors can often find creative ways to bypass these rules and access restricted information.

If these risks sound familiar, that’s because they are — prompt injection is akin to SQL injection, while data poisoning is another form of tampering. The problem is that AI introduces these risks in unfamiliar forms that traditional threat modeling cannot predict. This requires a different mechanism altogether.

Threat modeling for AI systems

While building a threat modeling framework fit for AI does involve accounting for the many differences between traditional and agentic systems, it doesn’t necessarily mean starting from scratch. In fact, many of the most basic elements of classic threat modeling, such as creating a diagram of the components you’re modeling and prioritizing threats, can remain. Instead, what changes are both the nature of threats under evaluation and the AI-specific behaviors that allow those threats to emerge. Any successful AI threat model needs to be able to account for these.

Here’s a framework you can follow for building your own threat model for AI:

Step 1: Know the landscape

Start out by getting to know the structure of the AI system you’re modeling. But this isn’t about nuts-and-bolts infrastructure — it’s about understanding what each component in the system is designed to do, the relationships it has across your organization, the tools and data it can access, and the actions it can perform. 

You can begin this process by creating an inventory of agents and LLM applications, RAG pipelines, MCP servers, and any plugins or APIs your AI is using. With this roadmap in place, you can start paying attention to how data and actions move through the system. This includes user prompts and agent outputs, as well as data and instructions that move between the AI and connected tools. As you do this, try to ask whether there are any points in which malicious information or instructions could be entered into the system.

Finally, once you’ve established the boundaries of the AI system, it’s necessary to understand how it changes and evolves. Unlike traditional software, AI systems and workflows aren’t static. In the process of carrying out their work, they delegate tasks to other agents, go down conditional decision paths, link different tools together, handle dependencies, and so on. All of these need to be analyzed and understood so that you know how bad actors may take advantage of them.

Step 2: Evaluate risk with a structured approach

For good reason, classic threat modeling uses a variety of frameworks to identify and plan for threats. Doing so provides an ordered method for comprehensively evaluating a range of different threats across a complex environment. So although AI threat modeling may be different in many ways, this is why structured modeling frameworks can serve a useful purpose here as well.

Some frameworks are specific to AI. One of the best examples is MITRE ATLAS. Based on the MITRE ATT&CK matrix for traditional cyberthreats, ATLAS provides organizations with an up-to-date knowledge base of AI-specific attack behaviors. You’ll find detailed descriptions of techniques like data poisoning, model evasion, and prompt injection, as well as implementation examples and suggestions for mitigation. All of this provides a useful starting point for organizations looking to build up a more formal threat model from the attack surface they’ve recently mapped.

For those searching for a more structured framework, there’s MAESTRO. Short for Multi-Agent Environment, Security, Threat, Risk, and Outcome, this framework is designed specifically for the unique set of threats and challenges agentic AI systems face. It’s built around a layered approach that takes apart the AI ecosystem into separate components:

• The agent ecosystem: Where agents interact with other apps and users.
• Security and compliance: An element of every other layer, this vertical touches on controls that are integrated throughout the system.
• Evaluation and observability: How agents get monitored for performance, accuracy, and other variables.
• Deployment and infrastructure: The architecture that the AI system runs on.
• Agent frameworks: This is the data and training that is used to build out the agents that make up the AI system.
• Data operation: Where data is processed, prepared, and stored for the AI agents
• Foundation models: The core AI model on which an agent is built. This can be a large language model (LLM) or other forms of AI.

Extended from the logic of more traditional threat modeling frameworks like STRIDE, the diversity of MAESTRO helps organizations address the more particular aspects of AI models, such as their unpredictable behaviors, the larger context they use to produce outputs, and the dynamic interactions and integration that they have. 

Step 3: Prioritize and take action

With the landscape mapped and threats identified, it’s time to move onto defense and mitigation. But before getting your hands dirty, you should take a strategic pause and consider the potential damage of each threat. 

Just as in classic threat modeling, not every threat will have the same likelihood or impact. There might be some that could be catastrophically damaging but highly unlikely, as well as others that would be much more likely to happen but would only incur minor damage. As detailed as possible, create an assessment of all of these threats and prioritize them according to their chances of occurring, the data or systems it would touch, and the blast radius of the damage. This will help focus your actions as you move forward.

Next, it’s a good idea to go through your AI system and apply least-privilege controls. Overly permissive access is one of the most likely ways for bad actors to take advantage of an AI system — and one of the most straightforward to fix. Just audit permissions by system and action type, associated task, and role. Ensure that agents only have access to the data they need and can modify files, execute code, or carry out requests according to their specific purpose.

Finally, start pressure-testing your model. Use adversarial testing and AI red teaming to run realistic attacks on your AI system so that you can understand how well your defenses hold up and find any hidden vulnerabilities. This should cover the full scope of identified threats, including prompt injection, data poisoning, MCP manipulation, data leakage, agent collusion, model extraction, and whatever else you ranked during prioritization. The ultimate goal here isn’t just to test out individual processes, but understand how the entire AI system and stack responds to threats.

Step 4: Continuous monitoring

Even more than other systems, AI environments are constantly evolving and changing — and so are the threats they face. This means any threat modeling you do is not static, but something you continually practice and adapt alongside your system. As models get updated, new tools get connected, and new capabilities emerge, so too will you need to monitor and evaluate how your AI system responds.

Make AI Threat Modeling a Cornerstone of Your Defense

As AI continues to proliferate and embed into businesses, so too are attacks. In just the past year, according to IBM, AI-related attacks have increased by 56 percent and add roughly $1 million to the cost of each breach. This kind of growth means that the threats and vulnerabilities your AI systems face aren’t something you can just respond to — you have to get ahead.

Threat modeling has long been a cornerstone of software development, and it’s now time to make it one for AI systems as well. There’s no better way to gain an understanding of the environment and behaviors of AI, the complexities of its threat landscape, and the strategies you must put in place to protect it from attack. This is essential not only for the integrity of your AI, but also for the entire risk profile of your larger business.

Looking for a good place to get started? Whether you want to learn how to keep your AI system secure or use AI itself to fight off threats, Cybrary’s got you covered. Check out our entire catalog to learn what we can do for you.

‍