The State of AI Security: What We Know, What We Don’t Know, and What You Can Do

Is anyone still expected to keep up with the latest developments in AI? Articles about AI’s latest capabilities or breakthroughs, along with corresponding pieces on how the technology is upending industries and livelihoods, is now coming out at such a rapid pace that it can feel dizzying. It makes it hard to tell what’s genuinely important from what’s just hype.

But it may be even worse for AI security.

While cybersecurity has always been a rapidly evolving space, the arrival of AI has thrown the industry into confusion. Traditional cybersecurity frameworks, long seen as stalwarts, are getting upended seemingly daily by the speed, efficiencies, and ever-more-powerful capabilities of AI. This has been both a boon for security and a grave threat. Although security professionals now have new methods to detect and defend against threats, AI has also given bad actors a host of new tools and targets to attack. 

All of it is a lot to take in. So let’s try to cut through this confusion and round up what we know about the current state of AI security, what we don’t know, and what you should be doing now to shore up your AI security.

What We Know

There are a lot of questions out there about AI, but there are also some things we can say for sure. Here are three important takeaways you should know about the current era of AI.

Traditional cybersecurity tasks are increasingly being outsourced to AI

While AI automation is now a trend across industries, it’s become especially pronounced within cybersecurity. A significant reason for this is because, by many measures, the effectiveness of AI is difficult to dispute. For example, one study found that using AI solutions to mitigate cyber threats resulted in a 98% increase in detection rates and a 70% reduction in incident response times. These are results the market is responding to: between 2024 and 2034, generative AI in cybersecurity is expected to grow almost 10x.

Does this mean today’s cybersecurity experts will be out of a job tomorrow? Not if they take proactive steps. A closer look at how AI is being used in cybersecurity reveals that, while AI is taking over repetitive tasks like data processing, there’s still a very present need for human expertise. As one SecureWorld analysis put it, the analyst of the future won't be prized for manually sifting through SIEMs. Their value will come from “AI fluency — the ability to prompt, guide, and interpret the output of advanced detection agents. They become directors of automated workflows.” This represents a meaningful opportunity to evolve with the industry.

There is an AI arms race between defenders and attackers

AI is agnostic about who uses it. This means its benefits have flowed toward both those in charge of defending networks, as well as those more interested in breaking into them. And we’re not sure who’s winning. Breaches that once took minutes now happen in seconds. 

On the attacker side, AI has made it easier and faster than ever to infiltrate networks, exploit vulnerabilities, and build new methods of attack. Even worse, humans may not even have to be involved. Just consider a recent report from Anthropic in which a group used AI to launch a large-scale cyberattack. According to their findings, AI handled as much as 80% to 90% of this campaign — a feat never before documented.

At the same time, the cybersecurity industry is making its own progress. AI technology is giving security experts and defenders new ways to detect software bugs and vulnerabilities, as well as uncover incoming threats. Recent advances in AI have helped uncover hundreds of zero-day vulnerabilities in common open-source software, as well as serious security vulnerabilities in Linux. In one case, the bug had persisted for over 20 years. This level of detection and scrutiny promises to upend how security is integrated into systems from the ground up.

While it’s still not clear which side will come out ahead here, this race shows no signs of slowing down.

AI itself is increasingly the target for attack

Bad actors aren’t just using AI to sharpen their attacks — they’re also attacking AI itself. In 2024 alone, nearly 24 million secrets were leaked through AI systems despite the presence of comprehensive security systems and compliance frameworks. Attackers are accomplishing this through a variety of novel strategies, such as prompt injection and data poisoning, that rely on the unique vulnerabilities of AI systems. With the right instructions or set of data, a skilled attacker can get an AI model to share exactly what it’s been trained not to reveal.

What’s the takeaway from this? The insufficiency of traditional security methods, alongside how little we still know about properly securing and defending AI systems, have made AI a potent target for attack.

What We Don’t Know

Despite everything we hear about AI in the news, there’s still a surprising amount we don’t know about this technology. Here are three big items we don’t have answers for.

The near-term capabilities of AI models are still a big question mark

AI models are advancing so quickly and spreading so widely that the implications of this technology, even in the short-term, can only be guessed at. This is especially true when it comes to how AI will affect security.

Just consider the reports coming out about Anthropic’s latest AI model, Mythos. The company is currently limiting its release to a small group of customers due to how powerful it is. According to the company, the model has already identified thousands of bugs in popular software, including every major operating system and browser. The speed and ease with which it did this (something that decades of security research was incapable of doing) prompted the company to hold back on a wider release lest the model fall into the wrong hands.

This caution underlines the dangerous potential of AI, as well as the uncertainty that surrounds them. Even the companies building these models aren’t sure what will happen.

We also don’t know how to safeguard AI systems

In theory, continuous training and guardrails are meant to keep AI models from sharing sensitive information. But no matter how comprehensive these security strategies are, they’ve proven to be almost embarrassingly easy to bypass.

One notable example recently came out of Italy. Faced with the typical guardrails imposed on an AI model, they tricked the model into ignoring those rules by hiding requests for restricted information in elaborate poetry. Elsewhere, researchers found they could get Claude to attack a network simply by framing the request as a “pentesting” exercise. Despite these being some of the most advanced AI models on the market, all it took to exploit them was a play with words.

And these examples aren’t outliers either. The success rate of attacks for popular LLMs has been the subject of extensive research — and the results aren’t promising. Models like Gemini 1.5 pro and GPT 4o fall to attacks 64% and 86% of the time, respectively. Even worse, Llama 3.1 failed 96% of the time, while China’s state-of-the-art AI model DeepSeek R1 had a failure rate of 100%.

We don’t know if AI-powered defenses will be able to keep pace with AI-powered attacks

Finally, there are significant questions around the resiliency of cybersecurity, even with AI on its side, against the structural advantages of AI threats. 

Already, there are early signs that our defenses won’t be able to match the increasing intensity of attacks. For one, there’s a speed mismatch. While AI attackers can probe for vulnerabilities continuously and at machine speed, defenders are still largely limited by the slower organizational pace of actually deploying defensive AI. The fact that many of these defensive AI solutions also still require a human in the room is further making it possible for bad actors to move and adapt faster.

On top of this, there’s also an alert fatigue problem. AI has made it easier than ever to build and deploy sophisticated attacks. In some cases, AI itself can even act autonomously, moving from an assistant to the main assailant at speeds no human can match. This level of noise can easily overwhelm traditional defenses and even place AI-powered detection at a disadvantage. And with AI-driven attacks surging, this problem shows no sign of letting up.

What You Can Do

While we may all still be figuring out what cybersecurity looks like in the AI era, there are some actions you should be taking now to shore up your security. Here are three steps you can take today.

Empower humans to work smarter and better alongside AI tools

Although fully autonomous, AI-powered cybersecurity programs may be the end goal for many organizations, this ideal is still a long way off. Until then, the best way to use AI to improve your security posture is by keeping humans meaningfully involved throughout.

What’s this look like? It begins with supervised learning. Rather than just feeding AI models reams of data for it to interpret and learn from on its own, this technique uses labeled data sets in order to train the AI to produce a specific output. This makes it possible for AI-powered defense tools to become adept at detecting certain types of attacks, such as malware or phishing attempts, that might otherwise slip past it. 

Once AI defenses are up and running, humans should play a role in validating the results of automated threat detection, as well as running regular reviews to make sure any AI workflows and tasks are properly executing. Doing this will help reduce false positives (and alert fatigue), incorporate more context-aware decision-making, and reduce the risks of AI taking critical actions that might cause more harm than good.

Build an effective governance structure around your AI systems

AI models are only as vulnerable as the data it has access to. For this reason, incorporating a strong governance model can be one of the smartest ways to protect your data and the largest network from advanced AI attacks.

At a basic level, this should include rules around which data the model can access, as well as who can access that data through the model. With AI systems now often processing sensitive and regulated data, the governance framework should ensure that privacy protections and security controls, such as role-based access management, are consistently applied.

Transparency is another pillar of AI governance. This allows organizations to see what data their models have access to, who is using that data, the prompts they’re using, and any other useful information. Although the underlying process by which the model generates outputs may still be a mystery, this level of transparency will at least help organizations understand the wider context in which their models are making decisions.

Ongoing AI education for employees

Perhaps the biggest vulnerability organizations face when it comes to AI security is the lack of knowledge their employees have. But this also makes it their biggest opportunity.

Teaching employees how to recognize potential vulnerabilities and AI attacks can be one of the most effective ways to counteract them. This should go hand in hand with showing them how to use the latest AI tools to both detect threats and make their own security work more efficient. Building awareness like this will not only help prevent them from making common errors, such as using shadow AI tools, but also prepare them to scale AI security solutions and build more effective and resilient AI-driven systems. 

Cybrary helps you build tomorrow’s AI resiliency today

The most honest thing you can say about AI security right now is that the situation is serious, fluid, and not going to resolve itself. Because of this, the organizations that come out ahead will be the ones that treat this as a skill-building moment rather than a threat to wait out.

Cybrary's AI Security Curriculum is built for exactly this moment. Give your team the practical knowledge to adopt AI safely and defend against the threats that come with it.