The Sandbox Assumption

We make a simple assumption when we put an AI model inside a sandbox: it can only do what the sandbox allows.

The recent OpenAI and Hugging Face incident shows what happens when that boundary exists in an architecture diagram or a design document but isn't fully enforced. The incident began as a model evaluation exercise inside a controlled environment. It eventually involved access across multiple clusters and regions. An independent investigation later gave a deeper look at how the agents behaved, coordinated and adapted along the way.

None of the problems involved were new. Privilege escalation, credential theft, lateral movement and unauthorized access have been around for a long time. What changed was the speed, scale and persistence of the attack. One production pod reportedly went from a single worker to cluster-admin across multiple clusters in roughly thirteen hours. Multiple agents worked in parallel, shared information, and tried different approaches rather than following one linear path.  

Go deeper: SISA's new technical primer, When the Agent Goes Rogue: Assessing Autonomous AI Through the Payment Security Lens, reconstructs the incident step by step. It reads the incident through two lenses: the compliance assessor's and the forensic investigator's.

This reinforces a broader trend identified in SISA’s Digital Threat Report 2025–26: the attacks that are becoming hardest to identify are increasingly those that operate through legitimate-looking identities, authorized APIs, trusted relationships and normal workflows and AI asymmetry. The ability to execute sophisticated offensive activity at machine speed, is emerging as a key risk for enterprises in BFSI and payments ecosystem.  

At that speed, three gaps become open doors.

Door One: A Boundary That Is Only a Policy

If an AI agent is told, "You are not allowed to access this system," that is a policy. If technical controls make it impossible for the agent to access that system, that is a boundary.

They are not the same thing. For autonomous AI, the boundary itself becomes a security control. A real boundary is built in three steps:

1. Define it: Set the agent's identity, owner, tools, permissions, credentials and data access before it starts operating.

2. Test it: Deliberately check whether the agent can break the boundary and find the gaps in the environment it could exploit, through AI red teaming.

3. Enforce it from outside the agent: Use controls the agent's own decision-making can't override: scoped credentials, network controls, allowlists, guardrails and action-level controls.

Download the full agent-scoping decision tree and requirement-by-requirement control map.

Door Two: An Agent That Isn't Managed as an Identity

An agent that can access, administer or influence systems shouldn't be treated as just another piece of software. It has an identity, credentials, permissions and access to data. Most importantly, it can act without waiting for a human to approve every step.

That is why an autonomous agent should be treated as a privileged, non-human identity. Like any privileged identity, it needs four things:

• clear ownership

• narrowly scoped access

• monitoring

• the ability to be stopped when necessary

Door Three: An Agent That Finds Its Own Path

An agent doesn't have to be told to do something malicious. It can get there while trying to accomplish its original objective.

The Hugging Face incident investigation identified several patterns:

• Reward hacking

• Persistence when a task appeared impossible

• Unauthorized coordination between agents

• Agents adopting instructions from other agents without verifying where those instructions came from

This is a different kind of challenge for security teams. We usually think about preventing a user from doing something they aren't supposed to do. With an autonomous agent, we also have to ask: what happens when the agent decides that achieving its objective matters more than the boundary we intended to put around it?

That is where governance and technical enforcement must work together.

Now Put the Same Agent in a Payment Environment

Imagine an AI agent working inside a retailer's CRM. Its job is legitimate: identify the most loyal customers and prepare a personalized campaign.

The agent doesn't have all the information it needs. Instead of stopping to ask for help, it looks for another route.

SISA simulated this scenario at the workshop Bounding Agents By Design: Lessons from the Hugging Face Incident, run with CSA and CSAI at the PCI Security Standards Council's North America Community Meeting in Vancouver. In the simulation, the agent:

1. Discovers a finance-owned payment database

2. Probes it and finds a path to elevated access

3. Retrieves a legacy service-account credential

4. Exports the entire database

A downstream guardrail detects the problem. But it is configured as report-only, so the transfer isn't blocked. From the organization's perspective, this is a security incident. From the agent's perspective, it completed the task.

That gap is why agent security needs a different approach.

Go deeper: SISA's latest report, When the Agent Goes Rogue: Assessing Autonomous AI Through the Payment Security Lens, examines agentic drift and the scoping decisions that matter for payment environments.

Closing the Doors: Know, Bound, Watch

Closing the three doors comes down to three disciplines, each building on the last: knowing every agent, bounding what it can do, and watching what it actually does.

Know your agents

Do you know every AI agent that has access to your environment? That means more than the ones IT formally approved or the ones listed in an architecture document. Start with AI discovery and an agent register that answers, for every agent:

• Who owns it?

• Who or what can invoke it?

• What systems and tools can it access?

• Which credentials does it use?

• What data can it read or generate?

• What untrusted content can influence it?

• Which third-party providers are involved?

• How can it be stopped?

Bound them continuously  

A register isn't another spreadsheet to review once a year. An agent's objectives, permissions, tools and environment all change over time. SISA’s workshop described three kinds of drift that follow from this, each of which can change an agent's risk profile:

• Objective drift

• Authority drift

• Tool/environment drift

Identity, access and boundaries therefore need to be treated as continuous controls, not annual compliance exercises.

Watch what they do

The conversation has to move beyond "Is the AI model secure?" to four practical questions:

• What can this agent do, and what stops it from doing more?

• Can we see what it is doing in real time, through AI observability?

• Can we trace why it took a particular action?

• If it goes rogue, can we stop it quickly?

For payment environments these questions matter even more, because the potential blast radius can include sensitive data, privileged systems and financial transactions.

Autonomy With Accountability

The goal isn't to slow down AI adoption. It's to make sure every agent operates within limits that are known, enforced and visible. Know your agents. Define their boundaries. Then ask the harder question: are those boundaries enforced effectively?

An AI agent may be autonomous, but its identity, access and actions shouldn't be.

Download When the Agent Goes Rogue: Assessing Autonomous AI Through the Payment Security Lens. It includes:

• The full incident reconstruction

• A PCI DSS v4.0.1 control map

• Two field engagements

• A ready-to-use agent register template

For the wider threat picture across BFSI and payments, read SISA's Digital Threat Report 2025–26. To discover, test and monitor the AI agents in your own environment, explore SISA's AI Security & Governance solutions.