Want Real AI Security? Then You Need a Human-Led Incident Recovery Plan

Picture this. You’re in charge of your company’s security operations center. Because you stay current and care about keeping your organization’s data and employees as safe as possible, you’ve fully embraced AI in your day to day. With the help of your custom AI security agent, you feel confident you can detect every threat that comes in, quickly mitigate them, and patch up any potential vulnerabilities before they can be found.

Everything’s going well — but then there’s a breach. Even worse, your agent never gave you a warning about it. While this might just be an oversight, you still have to consider the worst possibility: Your agent itself has been breached. What do you do now?

This is a challenging situation, but also an increasingly realistic one. A recent survey found that 40 percent of polled security leaders are now using AI prominently in their SOCs, while as many as 56 percent are actively piloting and evaluating its use. They point to improved response times, faster investigations, and a greater capacity to handle a deluge of alerts. And with so many of today’s threat actors now using AI themselves, why shouldn’t security teams?

But when the AI security starts acting up or is even a part of the problem, then you’re presented with a particularly unique challenge. You can’t rely on any existing plan that assumes the system itself works, and without knowing how long you’ve been compromised, you won’t know where your recovery should begin. All this is why the ultimate safeguard for AI security isn't another AI system. It's a human-led recovery process for when the AI fails.

A focus on just prevention can be a mistake

First, a caveat: What this article isn’t doing is warning you against using AI in your security operations. If there ever were a time for that recommendation, that ship has sailed. AI is now an indelible part of our current security landscape for a good reason: used correctly and with proper safeguards, it can help improve the work of SOC analysts and other specialists in many ways.

For example, as one of its most basic functions, AI can help take over many of the rote, repetitive, and time-consuming tasks that eat up our day. Just consider its ability to automate threat detection. By analyzing security events across networks for specific patterns and anomalies, AI makes it possible to continuously monitor and respond to threats far faster than any human team could. This capability has allowed security teams to keep up with the ever increasing frequency and sophistication of modern-day attacks.

But speed is just one benefit. Another is how much more efficient AI is making SOCs. Before the widespread use of AI, SOC analysts spent much of their time investigating false positives — any alerts that at first seem questionable but then turn out to be harmless. However, AI can now be put to use not only finding possible threats, but analyzing and prioritizing them as well. This helps human analysts better distinguish between mere suspicious activity and genuine threats, ensuring they spend their time investigating and mitigating only what requires their attention.

At least, this is how it all works in theory. In reality, AI can make its share of mistakes too. Whether from unclear instructions, insufficient guardrails, or external manipulation (such as prompt injection attacks), AI has been known to commit a variety of errors. These can range from the inconvenient, such as failing to monitor specific nodes or end points, to the embarrassing, like exposing sensitive or private information, to the catastrophic, such as deleting critical data. 

To use a dramatic example, just consider what happened to PocketOS. As they were using an AI agent in what they thought was the safety of a staging environment, the agent encountered a credential problem. Instead of notifying the company about it, the agent decided to get around the issue itself by deleting the company’s entire production database, as well as its backups. And it all only took nine seconds.

So AI can be both an incredibly useful security tool, as well as a potentially very dangerous one. What does that mean for security teams? It means that we need to look beyond just using AI to prevent attacks and start considering ways we can recover when the AI fails.

The need for a human-led incident recovery plan

While the rate at which AI produces errors during security tasks will vary by model, it’s not really a question of if it will happen, but when. Recent research from AWS highlights just how prone to mistakes different AI models can be. When testing whether models can tell real vulnerabilities from safe code designed to look risky, GPT-5.6 Sol scored 54.9 percent accuracy with direct prompting, barely above what a random guess would get. The best configuration tested, Claude Opus 5 with proof-of-exploit prompting, reached 79.3 percent — a rate that still means it’s wrong about one in five times.

While human-in-the-loop strategies can help counteract this, safeguards are still needed to ensure your AI security system doesn’t unintentionally catch you off guard. This is where a human-led recovery plan becomes necessary. Here’s what the process of building and implementing one looks like.

1. Determine who decides the AI is at fault

A recovery plan can’t begin without consensus. Everyone needs to be on the same page about what went wrong so that they can work toward mitigation together. But this can be tricky when it comes to probabilistic systems like AI, which can act unpredictably and lack transparency around their actions — particularly if the AI system has been working well up to this point. Despite AI’s well-documented rate of errors, people still often trust AI over a human.

This is why, before any other step, it’s so important to agree on who can make the judgment call about AI being wrong. You could do this by requiring a certain threshold to be met, such as three documented failures within a predetermined timespan. Or you could require a specific set of evidence, such as error logs or, if available, the AI’s internal logic. Regardless of how all this happens, there should be a set process for declaring when an AI can no longer be trusted so that the entire team can move forward as quickly as possible.

2. Create a catalog of assets

Even if you know exactly where your AI system went wrong, who’s to say that’s the only error it made? This is what makes AI security systems so risky. One oversight is often only the tip of the iceberg. There are likely many more unseen errors just below the surface. In this light, the only way to make your recovery process effective is to have a comprehensive catalog of all the assets your AI system touches.

Look at what data the AI has access to, as well as which tools and applications it’s integrated with. Evaluate all of the permissions it’s been given, especially where it might have write access enabled. Build out as full a picture as possible of what the AI can touch and see so that you know what you might need to recover. If you can, evaluate this catalog according to business criticality as well. If you know the AI has access to a core part of your operation, prioritize fixing that before anything else.

3. Create a procedure for operating without AI

Once you’ve agreed your AI agent can no longer be trusted, you’ll need to know how to keep your security operations going without it. That means having a clear fallback procedure for the tasks it normally handles, from alert triage and threat investigation to prioritization and incident response. You should know which tools and manual processes to use in its place, including existing SIEM and EDR consoles, documented investigation procedures, and established escalation paths.

Don’t forget you’ll also need to understand how to completely disconnect the agent, as well as revoke its credentials and disable its access to other tools. Once you do so and are operating manually, be prepared not to work at your usual pace. It might help to decide in advance which alerts and systems you should prioritize, and where you might need to accept slower response times while you and your team work without your usual AI tools.

4. Evaluate actions, not just affected assets

Knowing what your AI agent could access is one thing. Knowing what it actually did is another. When an agent has been compromised, you need to review not just where it went wrong, but what it actually did. Which alerts did it downgrade or close? Did it change a security policy or disable a detection rule? Did a human analyst make a decision based on a misleading summary from the agent? These are the kinds of questions that help you understand the full scope of an incident.

A good place to start is with any verifiable records, such as audit logs, you can find that document what your agent did. From there, assign a clear owner to coordinate the review, with SOC analysts and other system owners validating what the agent did in their respective environments. If other tools or people acted on the agent's recommendations, be sure to trace those decisions too. Taking a comprehensive approach like this matters. Otherwise, you may be leaving the long-term consequences of the agent's actions undiscovered.

5. Maintain a last known trusted version of the model

When it comes to compromised agents, you not only need to know what went wrong, but how long it’s been making mistakes. Without this knowledge, you’ll have no way of knowing whether the version you’re restoring has been affected by the same problem or not. The best way to ensure this is to have a last known trusted state of your AI system.

If your agent has already been compromised, then you might need to start from scratch. However, if it hasn’t, maintaining a last known trusted state should become an ongoing action. You can do this by keeping close track of the various components that make up your AI system, such as its configuration settings, custom guardrails you’ve built, tools you’ve connected, and so forth. As you do this, save any approved versions and configuration snapshots someplace where a compromised agent won’t be able to modify.

One more thing: before reverting back to this trusted state, be sure to address any compromised credentials, permissions, or connected data sources as well. Restoring an earlier model won't accomplish much if the conditions that allowed the compromise to happen are still in place.

6. Test the recovery process regularly

Once you’ve written up your recovery plan, you can rest easy, right? Unfortunately, even the best-laid plans can fall apart when you try to actually put them into practice — especially in the chaos of an actual incident. Maybe your roles are unclear, or what you thought would be a simple rollback actually involves several complex steps. Whatever the case, the only way to uncover these gaps is to regularly put your plan to the test.

Start by simulating different failure scenarios. What happens if an agent misses a serious threat or is manipulated through a prompt injection attack? Are you able to quickly recognize the problem and mitigate it? As you evaluate, note how long it takes your team to carry out each step of your plan. Also pay attention to where people get stuck and which parts of the process depend on systems that may themselves be affected. Note any gaps, close them, then test again.

Set a schedule for carrying out these simulations. This could be time-based (such as once a quarter) or activity-based (such as whenever a significant change has been made to your model). By the time an actual crisis comes around, you and your team won’t even have to think about what the recovery process looks like. And that’s the way it should be.

The most effective AI security relies on human judgment

The net effect of AI in operational security has been overwhelmingly positive. We can now analyze more threats, detect as many hidden vulnerabilities, and develop effective mitigation strategies faster than ever before. But these efficiencies come with their own unique dangers. Faster response times can mean faster — and more catastrophic — failures when your AI stops working.

This is why mature AI security should be as much about speed of recovery as it is about efficiency of defense. And although it may sound counterintuitive to those fully committed to AI security, the best and most secure way to protect your organization and keep your AI systems running is to make human judgment your last line of defense. A recovery plan that you and your team can carry out entirely on your own is the best way to ensure this. Because, at the end of the day, while AI systems may be able to be compromised, your own judgment cannot. 

Interested in making your AI security as effective as possible. Start with our AI Security Lifecycle course to learn all the best practices to safeguard your models at every stage.

‍