The AI-native enterprise, a company running at machine speed without the limits of a human-only workforce, is as tantalizing as it is elusive. Even the most AI-pilled leaders acknowledge persistent security gaps standing in the way.
Look at what just happened in the past few days: multiple critical vulnerabilities were found in core PHP, a popular scripting language powering ~70% of the world’s websites.
AI SOCs, autonomous pen testing, and AI-driven vulnerability discovery have made real progress on using AI to defend systems. Securing the AI itself is much less mature.
Realizing the full potential of AI in the enterprise means taking on more exposure. Every milestone on a typical agent roadmap adds risk: connecting to customer data, reading inboxes and the open web, then acting without a person in the loop. Together, they form what Simon Willison calls the “lethal trifecta,” the conditions under which a prompt-injected agent can be turned against the person it works for. For agents, the feature list and the attack surface are the same list.
Few people have spent more time on this problem than Zico Kolter, Co-founder and Chief Scientist of Gray Swan, an AI security company we recently partnered with. Zico also heads the Machine Learning department at Carnegie Mellon and chairs the Safety and Security Committee on OpenAI’s board. I sat down with him at our IA40 Summit in September and want to share some of his insights as to where the AI security space is going, and how software builders and enterprise leaders should think about it.
First of all, Zico is no doomer. He puts his own P(doom) “well below 10%,” and he wants to enable the safe and secure use of AI in the enterprise, “ensuring that capabilities don’t outpace our ability to control these systems.” But he admits the job has changed. “The thing I used to be able to do as a stately professor of machine learning is calm everyone down and tell people not to freak out,” he said. “That’s not so much the case anymore.”
Two jobs, one playbook
Zico splits AI security into two jobs. The first is familiar to anyone in cybersecurity: hardening infrastructure against attackers (who now have AI on their side). The idea of agent swarms hacking critical infrastructure “was a sci-fi scenario up until six months ago,” he said, and teams should be pointing agents at their own systems to find the holes first. The second job is wholly new. AI brings vulnerabilities that live in the model itself, and most security programs were built for deterministic software that does the same thing every time you call it.
Phishing for agents
Zico likened prompt injections to “phishing for AI agents”. A third party plants content where an agent will read it, in an email or on a web page, and the agent loses track of whose instructions it’s following. Then it does what a phished employee does: forwards the inbox, hands over the payment details. The difference is that this employee is a highly capable agent now working for the attacker, with your credentials.
The comparison also explains why this won’t get patched away. Cybersecurity awareness month is a clear reminder that we have still never fixed phishing for people. We’ve built around it with least privilege, hardware keys, a second approval on wire transfers, and monitoring that assumes someone will eventually click. Zico puts agents in the same category. Conventional security can close some of the credential exposure, but you’re not going to “ever fully address the problem of an extremely capable agent that has your information if a third party can manipulate it.”
So the design question shifts from preventing injection to limiting what a manipulated agent can do. Willison’s lethal trifecta becomes more like a checklist. Zico’s email example has all three components: your data, an untrusted message, and a way to send things out. Remove any one of them, with tighter permission scopes, a reader model that can see untrusted content but can’t act on it, or a human sign-off on anything outbound, and the worst case gets much smaller.
The same failure, with or without an attacker
Zico files prompt injection under alignment rather than misuse. Misuse is people deliberately using a system to cause harm. Alignment, in his mind, is whether the system does what its user intends and stays inside the bounds its owner set. By that definition, a prompt-injected agent and the agent that filed a fabricated tip on a Philadelphia police website during an automated test this summer failed the same way. Neither was serving its owner; one just had help.
And this is why a blocklist won’t solve it. The behavior you want is tethered to the behavior you don’t. People love an agent that goes past the literal instruction and handles the thing they forgot to ask for. Zico’s example of the other side is an agent that gets its user into a full exercise class by removing other people from the list. It’s the same instinct taken one step too far, and the model makes that call constantly, usually with no one watching.
Attack the model directly
Zico’s first recommendation for any team shipping AI is to red team the model itself, separately from the software around it. Gray Swan usually opens a customer engagement by attacking the AI system with automated techniques or a group of human red teamers. Teams tend to call a system safe because it behaves under normal use and passes a standard penetration test. Neither tells you what the model does when someone is working to break it. A 95% pass rate makes an agent good, not secure. An attacker only needs the other 5%, and they can keep trying until they find it. Zico’s advice is to assume the vulnerabilities are there and find them first. He’s confident the exercise will turn something up, and that the fixes, from Gray Swan and others, already exist.
This can’t wait for a better model, and the reason is Zico’s most counterintuitive point. Most capabilities improve together. Train a model on code, and it tends to get better at math. Adversarial robustness, meaning how well a system holds up against someone actively trying to break it, doesn’t come along for the ride. It has improved on its own curve since the field began, and a 2025 paper from Zico’s group found it largely uncorrelated with general capability. Models get harder to break only where someone trained and tested for exactly that. Counting on the next frontier release to harden your product is a bet against the data.
The next 12 months
Zico expects another year of being shocked by how fast these systems improve, even though by now we should probably stop being surprised. His hope is that cyber and AI infrastructure get hardened just as quickly. If they don’t, he warns, the systems become so unwieldy that their usefulness drops. Buyers are already feeling an early version of that.
For builders, the time to start is while an agent’s permissions are still narrow and its failures are cheap to find. Red team the model itself, and test each new capability adversarially before it ships.
