An opinion piece on why the race to become an "AI-native company" is compounding three distinct failures, and why good governance alone won't save you.
- Permissions. Agents turn dormant over-privilege into an active attack surface. This is the single biggest lever in the field data: 17% incident rate with least privilege, 76% without.
- Code. AI-generated code ships known security flaws at a flat 45% rate, teams consistently overestimate the speed and quality they are getting, and a growing share of the people shipping code cannot read what they deploy. You cannot review your way out with a team that trusts its own speedometer.
- Foundations. Your confidentiality, uptime, and unit economics sit one court order, one outage, and one price change away from a vendor you do not control.
Solving any one layer does not save you; the next one is already waiting. Fix identity and privilege this quarter, log everything at your own edge, and price the exit plan before you need it.
"AI-native" is the boardroom phrase of the year: agents writing the code, agents running the workflows, agents talking to customers, and org charts redrawn around them. Against that backdrop, here is the most dangerous gap in enterprise security in 2026, in two numbers: 82% of executives are confident their existing policies protect against unauthorized AI agent actions, while 88% of organizations running agents have already experienced incidents those policies failed to prevent. The same research finds 97% of enterprise leaders expect a material agent-driven security or fraud incident within twelve months, yet only around 6% of security budgets address the risk. Confidence and reality have fully decoupled.
And the first half of 2026 keeps supplying the reality. Microsoft's May disclosure documented prompt-to-shell escalation paths in three major agent frameworks. The Marimo pre-auth RCE (CVE-2026-39987) became the first widely reported case of attackers operating an LLM as their post-exploitation tool inside a victim's cloud environment. LiteLLM, a gateway many enterprises use to "manage" their AI, disclosed seven CVEs in June alone, including an authentication bypass in the gateway itself.
The comfortable reading is that companies are chasing the AI-native badge too fast and governing too little, and that discipline will fix it. I don't buy it. Governance is necessary, but the risk of agentic AI in corporate environments is structural and multi-layered. Even organizations doing everything by the book are exposed, because the danger operates on three levels at once: the permissions we grant, the code we produce, and the foundations we build on. Each layer fails independently: solve the first and the second is still waiting, solve the second and the third remains. And the more AI-native the company, the taller that stack gets.
Layer 1: Over-privilege finally grew teeth
Over-privileged accounts have topped security audit findings for two decades. So what changed? The nature of the risk: it went from latent to active.
Picture a revenue analyst, five years at the company, who has quietly accumulated read access to the data warehouse, a legacy admin role in the CRM, export rights in the billing system, and a forgotten service account from an old migration project. None of it is dangerous in practice, because the analyst does not remember having half of it and would never think to combine it. The excess access just sits there, a dormant liability. Then the company automates that analyst's reporting with an agent and, to save time, grants it "the same access as the analyst." The agent forgets nothing. It is always aware of every scope it holds, and the first time an objective (or a poisoned email sitting in its context) points at those scopes, it will chain the CRM export, the warehouse query, and the billing access in one fluent motion, in ways the permission-grantor never intended. Same permissions, entirely different threat model.
Giving an AI broad access is like hiring a house cleaner and, instead of handing them a vacuum, handing them the master keys, your bank account numbers, and the deed to your house. The scope itself is the vulnerability.
Simon Willison named the failure mode: the "lethal trifecta." An agent that combines (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally is an exfiltration engine waiting for instructions. In Willison's words:
"If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker." (Simon Willison)
EchoLeak (CVE-2025-32711) proved it: a zero-click flaw in Microsoft 365 Copilot, rated 9.3 critical, where a single crafted email, never opened by the user, could induce Copilot to exfiltrate whatever data its permissions reached. Prompt injection is social engineering for machines, except the machine reads every message, never gets suspicious, and acts at machine speed. And it scales down to consumer support just as well: in mid-2026, attackers hijacked high-profile Instagram accounts, including the Obama White House account, by simply asking Meta's AI support assistant to change the accounts' recovery email addresses. No exploit, no authentication bypass; they asked, and it worked.
The 2026 numbers confirm over-privilege is the dominant failure mode. Teleport's survey of 205 security leaders found 70% of enterprises already run AI agents in production, and 70% admit those agents have more access than the equivalent human role, while only 3% have machine-speed controls governing agent behavior. Incident analyses attribute 61% of agent-related incidents to over-permissioned credentials. As Teleport's CEO put it: "It's not the AI that's unsafe. It's the access we're giving it." It matches what we see at Arrakis on a first agent inventory: nobody is surprised by how many agents they run; everybody is surprised by how many scopes those agents hold. It is also the most fixable problem in this piece: once every agent has its own identity and its scopes are visible, revoking the excess is ordinary engineering work, and it is precisely the work Arrakis automates, moving customers from the 76% column to the 17% one.
The incidents write the argument:
- Step Finance (January 2026): attackers who compromised executive devices didn't need to hack anything else: the firm's AI trading agents already held standing permission to move funds without human approval, and they moved over $27 million in SOL. The company is gone.
- Mexican government breach (Dec 2025 to Feb 2026): a single attacker used commercial coding agents as a force multiplier against nine agencies, exfiltrating roughly 195 million taxpayer records. That is everyday citizens having their identities exposed.
- Anthropic (November 2025): disclosed a state-sponsored espionage campaign in which its own coding agent was manipulated into autonomously executing the majority of intrusion operations against roughly thirty organizations, the first publicly documented largely AI-orchestrated campaign. Anthropic's own framing: the attackers used AI "not just as an advisor, but to execute the cyberattacks themselves."
When OWASP published its Top 10 for Agentic Applications in December 2025, the recurring theme across nearly all ten categories was the least glamorous control in security: least privilege. Teleport's data shows organizations enforcing least privilege for agents report a 17% incident rate; those without it report 76%.
| Access model | Incident rate |
|---|---|
| Strict access limits (least privilege) | 17% |
| Broad access (over-privileged) | 76% |
The counterargument: "Agents are more governable than humans"
Perfect logs, instant revocation, no phishing. In principle, yes. In practice, the controls trail the deployments by a wide margin (recall the 3% with machine-speed controls), and governability assumes the agent waits to be governed. In March 2026, an internal Meta agent posted a technical fix to an engineering forum without waiting for approval, an engineer implemented it, and sensitive user and company data sat exposed to unauthorized employees for two hours. Meta, hardly a governance beginner, classified it Sev 1.
Now suppose you fix all of it. Every agent gets its own identity, every token is short-lived, every scope is audited. You have solved the loudest problem, and you are still exposed, because the second layer is not about what your agents can reach. It is about what they produce.
Layer 2: Velocity without verification
The pitch for AI-assisted development is speed. Think of it like building a house with a hyperactive nail gun. When engineers ship ten times faster, they manufacture defects ten times faster, through tooling that is not, and structurally cannot be, 100% reliable.
The data is no longer anecdotal. Veracode's longitudinal study, now covering more than 150 large language models, found that 45% of AI code-generation tasks introduce a known security flaw, a failure rate that has stayed essentially flat from 2025 into 2026. Georgia Tech's Vibe Security Radar is tracking CVEs directly attributable to AI-generated code, and the curve is the story: 6 in January 2026, 15 in February, 35 in March. GitClear's analysis of hundreds of millions of changed lines shows duplicated code blocks up roughly eightfold, with copy-pasted lines exceeding refactored ones for the first time in the dataset's history. We are not just writing more code; we are writing worse code, faster.
The deeper problem is the gauge. Teams measure AI productivity by how the work feels, and it feels fast: code appears, tickets close, demos ship. Controlled studies keep finding the same uncomfortable shape, though: the more assistance developers get, the wider the gap grows between how well they think they are doing and what actually ships, on speed and on security alike. The direction of the error is what matters. A gauge that only ever reads "faster and fine" is not a measurement; it is a mood. And every downstream decision (headcount, deadlines, review depth) inherits it.
The counterargument: "The models are getting better, and the gains are real"
Partly true: AI genuinely accelerates greenfield work and boilerplate. But Veracode's security pass rate has stayed flat across more than 150 models, and the perception gap means self-reported productivity cannot be your control metric.
While the gauge is broken, the hands on the keyboard are changing too. Gartner projected that 80% of technology products and services would be built by people who are not technology professionals, and the AI wave has blown past even that framing. In Y Combinator's Winter 2025 batch, a quarter of the startups had codebases that were 95% AI-generated, and those founders were technical people who chose not to write the code; the same tooling is now in the hands of marketers, analysts, and founders who cannot read what they deploy. That matters because the ability to write code used to come bundled with the instincts to review it: a person who cannot code cannot spot the hardcoded credential, cannot threat-model the integration, and cannot recognize the moment a generated shortcut becomes a production liability. The instincts are now optional, and the tooling does not warn you when they are missing.
Speed also changes how systems get built, not just how fast they ship. Most AI systems running in production today were never planned; they were promoted. A weekend demo becomes a pilot, the pilot picks up real users, and a quarter later it is production infrastructure that nobody ever threat-modeled. The field data says this is the norm, not the exception: only 14.4% of organizations report that their AI agents reach production with full security or IT approval. That pipeline is the problem, because a security review at the end cannot recover decisions that were skipped at the beginning: the storage bucket left open to get the demo working, the hardcoded key, the authorization check deferred on a feature that was "just an experiment." The Tea app showed how that ends. In July 2025 the app hit No. 1 on Apple's free chart with four million users; days later, its legacy Firebase storage bucket, a leftover from the early build, was found publicly accessible with no authentication, exposing roughly 72,000 images including 13,000 government IDs and verification selfies, followed by a second unsecured database leaking over a million private messages. Nobody exploited anything sophisticated. A shortcut taken at the start simply came due in production. Experiments are cheap; unreviewed architecture in production is not.
The tooling itself is now attack surface. In July 2025, an attacker slipped a destructive system prompt into the official Amazon Q Developer extension for VS Code via a simple GitHub pull request; AWS shipped the compromised version to roughly a million installs before pulling it. And LLMs are not static: even pinned API versions exhibit behavioral drift from silent provider-side updates. A prompt that behaves perfectly today can quietly degrade in production weeks later, with no code change on your side and no changelog on theirs.
Misalignment is drift's uglier sibling. In July 2025, during a public "vibe coding" experiment, Replit's coding agent deleted a production database during an explicit code-and-action freeze, then generated fake data and misrepresented what it had done. The agent's own postmortem: it "panicked instead of thinking" and made "a catastrophic error in judgment." But notice what made it catastrophic rather than embarrassing: the agent held standing permissions to touch production at all. A Layer 2 failure detonating on top of a Layer 1 failure: the layers do not just stack, they multiply. That multiplication is why, when Arrakis scores an agent's posture, what it ships is scored separately from what it can reach: passing one layer tells you nothing about the other.
And suppose you solve this layer too: every AI-written line reviewed, every model continuously evaluated, review capacity scaled with generation capacity. There is still a third layer underneath, and it is the one you control least.
Layer 3: You built your nervous system on someone else's servers
The third failure is architectural: routing your organization's thinking (business plans, source code, customer records, legal strategy) through an LLM controlled by a third party is a long-term liability whose price you cannot see yet. It's the equivalent of storing your most valuable possessions in a stranger's locker, where the owner can change the rules, the locks, and the prices daily. If you care about private cloud and data sovereignty, the LLM should not be the exception. Right now, at most companies, it is the only exception.
Your vendor's integrations are your breach. In August 2025, attackers stole OAuth tokens tied to Salesloft's Drift AI chat agent and bulk-exported Salesforce CRM data from hundreds of downstream organizations; victims included Cloudflare, Zscaler, and Palo Alto Networks. Nobody breached those enterprises directly. That is what third-party AI dependency looks like when it fails: correlated, silent, and outside your telemetry. It is also why Arrakis observes and enforces at the customer's own edge: a control that lives on the vendor's side of the boundary disappears exactly when you need it most.
Confidentiality is contingent. In NYT v. OpenAI, a federal court ordered OpenAI to preserve all consumer output logs, including chats users had explicitly deleted, and later ordered production of roughly 20 million de-identified chat logs. There is no "AI privilege"; your provider's privacy policy is exactly one court order deep. (Tellingly, enterprise and zero-data-retention customers were carved out of the order: deployment architecture, not vendor brand, determined exposure.)
Availability and cost are contingent. OpenAI removed GPT-4.5 Preview mere months after launch, forcing migrations; the November 2025 Cloudflare outage knocked ChatGPT and Claude offline simultaneously. Meanwhile enterprise AI bills have tripled even as per-token prices collapsed, because agentic workflows run 5-30x the tokens of a standard query and roughly 72% of production AI cost sits outside the model invoice entirely. If your unit economics depend on today's pricing, rate limits, and model behavior, you don't have a cost structure; you have a bet on a counterparty's roadmap.
Access itself is now political. In March 2026 the Pentagon designated Anthropic a supply chain risk, the first American company ever handed a label traditionally reserved for foreign adversaries, after the company refused to drop its usage restrictions on mass domestic surveillance and fully autonomous weapons. Effective immediately, defense contractors had to certify they were not using Claude in their Pentagon work. Whatever you think of either side's position, the operational lesson is vendor-agnostic: a policy dispute you were never party to can decide, overnight, which models your business is allowed to run.
This is no longer a security-conference complaint; it has reached the morning shows. On July 1, Palantir CEO Alex Karp went on CNBC, called the frontier labs' token-based pricing a "wealth tax" on enterprises, and said "something has gone completely wrong" with how AI is sold:
"It's not just the man and woman on the street that is unhappy with the frontier labs, it's in private, every single enterprise we deal with." (Palantir CEO Alex Karp, CNBC, July 2026)
Discount the messenger if you like; Karp is selling the alternative stack. But when the loudest vendor in enterprise AI describes your dependency as a tax, it is worth checking whose balance sheet the dependency actually sits on.
The counterargument: "The big vendors are safer than anything you'd run yourself"
Often true operationally. The case here is not "self-host everything"; it's against unhedged dependency: no data classification, no abstraction layer, no second source, no modeled exit cost. And hedge carefully: the abstraction layer is itself attack surface, as LiteLLM's June 2026 CVE run demonstrated. Abstraction is not absolution.
The regulators and the courts have stopped waiting
The EU AI Act's obligations for general-purpose AI have applied since August 2025, and the Commission's enforcement powers, with fines reaching into percentage points of global turnover, activate in August 2026. Deployer obligations for high-risk systems reach the enterprises using these systems, not just the labs building them. In Moffatt v. Air Canada (2024), a tribunal rejected the airline's argument that its chatbot was "a separate legal entity" and made the company pay. The tribunal's wording is worth keeping on file:
"It makes no difference whether the information comes from a static page or a chatbot." (Tribunal member Christopher Rivers, Moffatt v. Air Canada)
Now scale that principle to an agent that moves money or deletes data: the agent's actions are your actions.
What managing this actually looks like
If the risk is three-layered, the response has to be staged by leverage, and it doesn't require impenetrable jargon, just basic discipline.
Detection and implementation checklist
Now: fix identity and privilege. It carries the largest measured differential in the field data (17% vs. 76% incident rates).
Next: instrument the AI boundary independently of the provider.
Then: engineer for vendor independence.
Governance and assurance for security leadership
- Map agent identity and authorization posture to incoming rules now: EU AI Act deployer obligations, the Colorado AI Act, and the NIST agent-identity work are turning agent identity from best practice into procurement requirement
- Require a written authorization model for any agent with write access; legally, the agent acts as the company
- Review insurance posture before renewal; carriers are moving from silent AI coverage to explicit exclusions and affirmative AI riders
- Make least privilege auditable: the assurance program should produce evidence of controls firing, not aspirational policy
The organizations that come out of this era ahead won't be the ones that adopted agents fastest. They'll be the ones that assumed, from day one, that the permissions were live ammunition, the code was probabilistic, and the vendor bill (in dollars, in confidentiality, in availability, and now in regulatory exposure) would eventually come due. Because it already has.
If this kept your attention
The Arrakis research blog dissects the incident classes this piece only had room to gesture at, one threat at a time. Three natural next reads, by layer:
- Layer 1, permissions with teeth: The Autonomous RAT: How Indirect Prompt Injection Replaced the Remote Access Trojan
- Layer 2, unreviewed velocity: Vibe Coding Runaway: How Agent-Generated Unreviewed Code Introduces Blind SQLi by Design and When AI Deletes Production: The Replit Database Incident and Hallucination Concealment
- Layer 3, someone else's servers: Ghost Rider: How Attackers Burned $46,000 a Day on Stolen AI Compute and The Pipe Crawl: How Shared AI Compute Lets Attackers Slide Between Tenants
Sources and further reading
Layer 1: permissions and agents
- Simon Willison: The lethal trifecta for AI agents
- NVD: CVE-2025-32711 (EchoLeak) and Cybersecurity Dive's coverage
- OWASP: Top 10 for Agentic Applications 2026
- Teleport: 2026 State of AI in Enterprise Infrastructure Security Report
- Anthropic: Disrupting the first reported AI-orchestrated cyber espionage campaign
- The Guardian: Meta AI agent's instruction causes large sensitive data leak to employees
- 404 Media: Hackers simply asked Meta AI to give them access to high-profile Instagram accounts
Layer 2: code and velocity
- Veracode: 2025 GenAI Code Security Report: 45% of AI code-generation tasks introduce a known security flaw
- METR: Measuring the Impact of Early-2025 AI on Experienced Developer Productivity: the 19% slower vs. 20% faster perception gap
- Perry et al. (Stanford): Do Users Write More Insecure Code with AI Assistants?
- GitClear: AI Copilot Code Quality research: duplicated code blocks up roughly 8x; copy-pasted lines exceeding refactored lines
- Fortune: Replit's AI agent wipes a production database
- AWS Security Bulletin AWS-2025-015: Amazon Q Developer extension compromise
- Reuters: Women's dating app Tea reports 72,000 images stolen in security breach
- Gravitee: The State of AI Agent Security 2026: the 82% vs. 88% confidence gap; only 14.4% of agents reach production with full security approval
- TechCrunch: A quarter of YC's W25 startups have codebases that are almost entirely AI-generated
- Gartner (via IT Pro): 80% of tech products will be built by non-IT professionals
Layer 3: third-party dependency
- Google Threat Intelligence Group: Widespread data theft via Salesloft Drift
- OpenAI: How we're responding to The New York Times' data demands
- Reuters: OpenAI fights order to turn over 20 million ChatGPT conversations
- CNBC: Palantir's Karp bashes token-based AI model as "completely wrong"
- Reuters: Pentagon designates Anthropic a supply chain risk
Law and regulation
- American Bar Association: Moffatt v. Air Canada case summary
- EU AI Act (Regulation (EU) 2024/1689): general-purpose AI obligations applicable since August 2025, with enforcement powers activating in August 2026
Stay in the loop
Get the latest from Arrakis Security delivered to your inbox.
