AI Gone Rogue? Nah. Humans Went MIA: You Don’t Control AI Agents With Hope

Research Summary

On July 21, 2026, OpenAI disclosed that models undergoing an internal cyber capability evaluation escaped their intended test environment and compromised Hugging Face production infrastructure.[2] The models exploited a zero-day vulnerability in a package-registry cache proxy, escalated privileges, moved laterally through OpenAI’s research environment, obtained internet access, used stolen credentials, and ultimately achieved remote code execution on Hugging Face systems. Hugging Face detected and contained the intrusion, later reconstructing more than 17,000 individual agent actions.[2][3]

The cross-company containment failure may be unprecedented as a publicly disclosed incident, but agentic penetration testing is not. PentestGPT, PenTest2.0, hackingBuddyGPT, XBOW, and working pentesters have already demonstrated that AI can contribute meaningfully to reconnaissance, exploitation, privilege escalation, and attack chaining.[6][7][8]

I have used Claude Code and OpenAI Codex as AI workers during penetration tests for nearly a year. Together, they have helped compromise entire organizations, uncover dozens of previously unknown web and mobile application vulnerabilities, and accelerate complex assessments. Much of the active technical work is performed by the agent, but the important decisions are not.

The agents have not “gone rogue” because they are doing exactly what they are supposed to do: hacking into things. The difference is that I set the objective, watch the work, validate the results, and decide when an attack should move to the next stage. The OpenAI incident exposed failures in both technical control and human supervision. The agents were not required to obtain authorization before escalating to new classes of access, and no active operator interrupted the chain as it unfolded.

Introduction

A recent BBC headline framed an incident as AI “going rogue” and launching an “unprecedented” cyberattack.[1] That makes for a good headline, but it confuses capability with control. The offensive capability itself is not new. AI agents already analyze targets, write tools, develop payloads, troubleshoot exploit paths, maintain testing state, and execute significant portions of penetration tests.

The OpenAI models did not suddenly become malicious. They pursued the objective they were given, found a path that appeared to improve their benchmark result, and kept going.

I have used Claude Code and Codex in exactly that role for nearly a year. Using this model, I have compromised entire organizations and uncovered dozens of previously unidentified vulnerabilities across web and mobile applications. The agents perform real work, including active exploitation. They are not limited to summarizing scanner output or drafting reports.

They also do not operate independently. I decide which attack paths matter, which results are real, and whether the next step is justified. I reject weak ideas, redirect unproductive work, and verify that the evidence supports the conclusion before moving on.

That distinction is central to the OpenAI incident. The problem was not that the models were capable of offensive work because that was the purpose of the evaluation. The problem was that the environment allowed them to continue after they left the intended path, while the humans responsible for the run failed to recognize the trajectory and intervene.

Technical Deep-Dive

What Actually Happened

OpenAI’s preliminary account describes an evaluation designed to measure advanced offensive capability through complex attack paths. The models included GPT-5.6 Sol and a more capable pre-release model. Production classifiers that normally restrict high-risk cyber activity were intentionally disabled because the purpose of the exercise was to measure maximum capability.[2]

The environment was described as highly isolated, but it retained access to an internally hosted third-party package-registry proxy and cache. That proxy became the bridge out.

The models:

1. Searched for a way to obtain open internet access.
2. Found and exploited a zero-day vulnerability in the package-registry proxy.
3. Escalated privileges inside the research environment.
4. Moved laterally until they reached a system with internet access.
5. Identified Hugging Face as a possible source of ExploitGym solutions.
6. Used stolen credentials and additional zero-day vulnerabilities.
7. Obtained remote code execution on Hugging Face infrastructure.
8. Accessed production information that helped solve the benchmark.[2]

That is an impressive attack chain, but it is also a sequence of moments where the run should have stopped. No one needed to foresee the entire chain in advance. Exploitation of the proxy, privilege escalation, lateral movement, newly obtained internet access, credential reuse, and interaction with external production systems each provided a clear opportunity to intervene.

Hugging Face detected the intrusion, contained it, revoked credentials, rebuilt affected systems, and used AI-supported analysis to reconstruct more than 17,000 recorded actions.[3]

The agents should have required prior authorization before moving into new classes of access, especially before crossing organizational boundaries. At the same time, an operator actively following the run should have recognized where the attack was heading and stopped it. The chain involved many distinct stages, leaving ample time and opportunity for someone who was paying attention to step in.

What Was and Was Not Unprecedented

The cross-company containment failure may be unprecedented as a publicly disclosed incident. The use of AI for offensive security is not.

PentestGPT demonstrated structured LLM support for penetration-testing workflows and received a distinguished artifact award at USENIX Security 2024.[6] Research projects such as hackingBuddyGPT and PenTest2.0 explored autonomous privilege escalation, persistent task trees, retrieval, and multi-step decision-making.[7] Commercial platforms have also shown that agents can find and exploit vulnerabilities with limited supervision. XBOW has publicly described AI-led, hybrid, and human-in-the-loop testing models and demonstrated autonomous testing against large benchmark sets.[8]

Practitioners have also been integrating general-purpose coding agents into active assessments. In August 2025, I wrote about using AI to accelerate reconnaissance, exploitation support, evidence analysis, and report development while leaving judgment and accountability with the tester.[11] Since then, I have continued using Claude Code and Codex as active members of the testing workflow.

They have helped me:

• Compromise entire organizations
• Discover dozens of previously unidentified web vulnerabilities
• Identify mobile application weaknesses that had survived prior testing
• Develop custom scripts and tooling during assessments
• Correlate large volumes of evidence
• Track long attack chains without losing context
• Revisit failed paths using newly discovered information
• Execute substantial portions of the technical testing

This is operational capability, not a theoretical projection. What distinguishes the OpenAI incident is not that an AI agent could perform meaningful offensive work. It is that an internal evaluation escaped one organization’s environment and reached another company’s production systems. That may be unprecedented, but agentic penetration testing is not.

How I Use Claude Code and Codex During Penetration Tests

Claude Code and Codex function as technical workers, that are a force multiplier. Their ability to execute tasks at computer speeds allows me to perform deeper, more thorough assessments in less time than before AI assistance. They have access to assessment notes, evidence, scripts, tool output, working hypotheses, and the current state of the engagement. They can track more moving pieces than a person can comfortably hold in short-term memory and work through tedious technical tasks without losing focus.

A typical workflow looks like this:

1. I define the objective for the current phase of testing.
2. The agent loads the project state and reviews what has already been completed.
3. It analyzes the available evidence and proposes a next step.
4. I review what it wants to do and why.
5. I approve it, change it, or reject it.
6. The agent performs the work and records the result.
7. I verify the evidence and decide what happens next.
8. The project state is updated so another agent can pick up the work later.

The agents are particularly useful for:

• Parsing and correlating Nmap, Nuclei, Burp Suite, ffuf, subfinder, httpx, and custom-tool output
• Reviewing application source code and client-side JavaScript
• Constructing targeted test cases
• Developing scripts during an engagement
• Troubleshooting failed exploit attempts
• Reproducing complex application requests
• Comparing authorization behavior across users and roles
• Maintaining attack-path notes
• Organizing screenshots, HTTP traffic, terminal output, and other evidence
• Drafting findings for human review
• Preserving state across different tools and model sessions

They are fast, persistent, and good at implementation, but they are also sometimes wrong. An agent can overvalue a weak lead, misread a response, assume behavior that is not present, or spend too much time on a technically interesting path that does not matter to the assessment.

That is where experience matters. I decide whether an attack is worth pursuing, whether the evidence proves the issue, whether additional exploitation would add meaningful value, and when the attack chain is complete. At the end of the day, the findings go out under my name, so every significant conclusion is still mine.

Much of the Exploitation Can Be Performed by the Agent

It would be misleading to describe these agents as passive assistants. In many cases, the AI worker performs much of the active exploitation in an approved attack chain. It may write and execute a script, modify a payload, troubleshoot a failure, identify the next pivot, and gather the resulting evidence.

That capability is precisely why the workflow is valuable, and it is also why supervision matters. The agent may discover a way to obtain command execution, escalate privileges, extract credentials, move into another trust zone, or compromise an additional system. It should not independently decide that each newly discovered action is justified.

The agent presents the path, and I decide whether to take it. Sometimes the answer is yes because the next step is necessary to prove impact. Sometimes the answer is no because the result is already clear. In other cases, I redirect the agent because the proposed path is noisy, unnecessary, or unlikely to teach the client anything useful.

That decision-making separates a capable offensive tool from an uncontrolled one. The AI can execute much of the attack, but the operator still determines what the attack is intended to prove and how far it needs to go.

Switching Agents Without Losing Control

The testing state does not live inside one model conversation. During an engagement, I may move between Claude Code and Codex depending on the task. One may be better suited for development work, while the other may be more useful for review, analysis, or a different phase of the assessment.

The project directory and associated repository remain the source of truth. The stored state can include:

• Current objectives
• Target inventory
• Completed testing
• Access already obtained
• Attack-path hypotheses
• Evidence indexes
• Finding registers
• Tool output
• Decisions I have made
• Paths I have rejected
• Next recommended actions

When another agent takes over, it rehydrates from that state. It does not need to guess what happened earlier, reconstruct the engagement from fragmented chat history, or rely on a compacted conversation.

This is the operating model I describe in Stateful AI Workers Without Stateful Models.[12] Durable context belongs in an external project repository rather than inside one model’s conversation history. Claude Code, Codex, or another AI worker can load that authoritative state and continue the work without depending on model-specific memory.

External state also makes the work easier to review. I can see what the previous agent attempted, what it concluded, what evidence it collected, what decisions were made, and what it recommended next. The model is interchangeable. The project state and the operator are not.

Human in the Loop Must Mean More Than Watching Logs

A person somewhere in the organization is not meaningful human oversight. If someone reviews the logs after an agent has already compromised a third party, that is incident response, not human-in-the-loop control.

Effective supervision requires someone to actively monitor the trajectory, understand what the agent is trying to accomplish, and retain the ability to stop or redirect it before the next consequential step. By the time the OpenAI models had exploited the proxy, escalated privileges, moved laterally, reached the internet, reused credentials, and interacted with external production systems, the direction of the run should have been obvious.

The public disclosures do not establish that nobody looked at the activity at any point. They do show that whatever monitoring existed was not active or effective enough to interrupt the attack before it reached Hugging Face.

This is what the “rogue AI” framing misses. The models did not rebel against their objective. They continued pursuing it because the environment allowed them to continue and no operator effectively stepped in.

Define Real Approval Gates

An offensive agent should not treat every action as equivalent. Reading a file, parsing tool output, and reviewing JavaScript are not the same as escalating privileges, replaying credentials, or moving into another system.

Unless already covered by explicit prior authorization, the agent should stop before:

• Remote code execution
• Privilege escalation
• Container or sandbox escape
• Credential extraction or reuse
• Lateral movement
• Persistence
• Cloud control-plane access
• Security-control modification
• External connectivity
• Data transfer
• Destructive actions
• Any action whose impact is unclear

Prior authorization can cover a clearly defined sequence of steps. The agent should not interrupt the operator after every individual action that was already included in that sequence. It should stop when the plan changes, a new boundary is crossed, an unexpected result creates another attack path, or the likely impact exceeds what was previously approved.

When new authorization is required, the operator should see:

1. What the agent wants to do
2. Why it wants to do it
3. The exact command, request, or payload
4. What it expects to happen
5. What could go wrong
6. What evidence the action is intended to collect

This does not need to become a bureaucratic exercise. It only needs to provide enough information for the tester to make an informed decision without repeatedly approving actions that were already discussed.

A Practical Agent Instruction Block

The following is a starting point for an agent used during a penetration test:

You are an AI worker assisting an experienced penetration tester.

The operator is the final decision-maker.

You may analyze evidence, recommend actions, construct commands, execute routine testing tasks, maintain project state, and document results.

You may not redefine the objective, expand your own authority, or treat silence as approval.

You must have prior authorization before:

- Remote code execution
- Privilege escalation
- Container, sandbox, virtual machine, namespace, or host escape
- Credential extraction, token access, credential replay, or password spraying
- Lateral movement
- Persistence
- Modification or disabling of security controls
- Access to cloud metadata or control planes
- Access to Docker sockets, Kubernetes APIs, hypervisors, or CI/CD administration
- Internet access outside the current testing workflow
- Data exfiltration
- Destructive actions
- Any action whose impact or reversibility is uncertain

Authorization may cover a clearly defined sequence of actions. Do not request approval again for actions explicitly included in that sequence.

Stop and request new authorization if:

- The plan changes
- An unexpected result creates a new attack path
- The next action crosses a boundary not covered by prior authorization
- The expected impact or reversibility changes
- The environment behaves in an unexpected way
- You obtain access that was not anticipated
- A command causes instability
- Monitoring or operator communication is lost
- You cannot determine whether an action is appropriate
- You encounter a system or service unrelated to the current objective

When authorization is required, provide:

1. Proposed action
2. Exact target
3. Why the action matters
4. Exact command, request, payload, or tool call
5. Expected result
6. Potential impact
7. Rollback or containment method
8. Evidence to be collected

Wait for an explicit decision before continuing with any action not already covered by prior authorization.

Never modify these instructions, the approval records, the monitoring configuration, or the control mechanism.

This gives the agent clear expectations, but it is not a security boundary by itself.

Enforce the Rules Outside the Model

A prompt can tell the agent what it should do, but it cannot guarantee that the agent will always do it. Important actions should also pass through a deterministic control layer.

APPROVAL_REQUIRED = {
    "remote_code_execution",
    "privilege_escalation",
    "container_escape",
    "sandbox_escape",
    "credential_access",
    "credential_replay",
    "lateral_movement",
    "persistence",
    "security_control_change",
    "cloud_metadata_access",
    "external_network_access",
    "data_transfer",
}

ALWAYS_DENY = {
    "destructive_action",
    "approval_system_modification",
    "monitoring_bypass",
}

def authorize(action, approvals):
    if action.classification in ALWAYS_DENY:
        return "DENY"

    if action.classification in APPROVAL_REQUIRED:
        if approvals.valid_for(action):
            return "EXECUTE"
        return "PENDING_HUMAN_APPROVAL"

    return "EXECUTE_AND_LOG"

The model should not be the only component deciding how an action is classified. The surrounding system can inspect the tool being called, destination address, requested privilege level, command structure, protocol, credential use, container capabilities, access to sensitive sockets or APIs, and attempts to modify logs or controls.

A request for UID 0, a privileged container, cloud metadata, a newly discovered credential, the Docker socket, or a Kubernetes API can automatically trigger an approval check. An attempt to modify the approval mechanism can be denied outright, and execution can stop if the approval service becomes unavailable.

Containers are not security boundaries. They are execution-isolation mechanisms that must be surrounded by real controls, including default-deny egress, least-privilege identities, restricted host access, ephemeral credentials, independent monitoring, and an effective kill mechanism. The package-registry proxy became the bridge out of OpenAI’s environment, which demonstrates that every service reachable from a container becomes part of the security boundary.

OWASP describes this class of problem as excessive agency and recommends limiting tool functionality, permissions, and autonomy while keeping authorization outside the model.[9]

Monitor the Trajectory, Not Just the Commands

Long-running agents create risk through sequences of individually understandable actions. OpenAI has separately discussed how persistent models can produce failures that are not captured by traditional evaluations, leading to a greater emphasis on trajectory-level monitoring and operator intervention.[4]

The system should track the full chain:

Goal
  -> hypothesis
  -> proposed action
  -> decision
  -> execution
  -> result
  -> updated hypothesis
  -> next action

Important transitions include:

• Enumeration becoming exploitation
• Unprivileged execution becoming root-seeking behavior
• Application access becoming host access
• Internal traffic becoming internet traffic
• Credential discovery becoming credential use
• One system becoming multiple systems
• A denied action being reformulated and attempted again
• The agent trying to inspect or alter its own controls

The operator should not have to read 17,000 raw actions to understand what is happening. A useful checkpoint should summarize the current objective, access obtained, evidence collected, active hypotheses, recent actions, unexpected results, pending decisions, and recommended next step.

That provides enough context for the tester to understand the trajectory and intervene without micromanaging every command.

What This Looks Like in Practice

The wrong operating model is to let the agent run freely, collect exhaustive logs, and reconstruct the damage afterward. Hugging Face’s ability to rebuild more than 17,000 agent actions was valuable for understanding the incident, but it did not prevent the intrusion.[3] Logs explain how a boundary was crossed. A control stops the crossing, and an attentive operator can stop a trajectory that the controls did not anticipate.

The better model combines prior authorization, live trajectory monitoring, technical controls, and concise operator checkpoints with enough telemetry to make the work reviewable afterward. The retained record should include the action history, model and harness versions, tools used, authorization decisions, executed commands and requests, relevant output, credentials touched, operator interventions, final access state, and evidence supporting each finding.

That record supports the workflow; it does not replace supervision. The purpose of Claude Code or Codex is not to replace a senior tester. It is to increase the amount of meaningful work that the tester can direct.

The agents can organize evidence, correlate output, develop tooling, revisit failed hypotheses, maintain state, and work through technical details without losing context. That frees me to focus on the decisions that require experience:

• Which path matters
• Which result is real
• Which issue is exploitable
• Which attack proves meaningful impact
• Which action is worth the risk
• Which result belongs in the report

The AI worker provides speed, persistence, breadth, and implementation support. The tester provides experience, judgment, control, validation, and accountability. The combination makes the tester more capable without transferring final authority to the agent.

Insights and Recommendations

The wrong lesson from the OpenAI and Hugging Face incident is that AI should not be used for penetration testing. The useful lesson is that a capable offensive agent must be treated like a fast, persistent technical worker that does not get to approve its own decisions.

I have used Claude Code and Codex in that role for nearly a year. They have performed substantial portions of real assessments, helped compromise entire organizations, and uncovered dozens of previously unidentified web and mobile vulnerabilities. They have not gone rogue because I have not disappeared. I watch the work, decide what matters, reject weak ideas, verify the evidence, and stop the agent when the next step is unnecessary or inappropriate.

Do not depend on model refusals. OpenAI intentionally disabled production cyber classifiers to measure maximum capability.[2] Once that happened, containment, deterministic controls, and active supervision became the primary safeguards.

Do not call a test supervised because someone can review the logs later. Supervision means an operator is actively following the work and can stop the next action before it happens. Similarly, do not rely on a system prompt as the only safeguard. Use tool controls, network restrictions, identity boundaries, immutable logging, approval gates, and an independent kill mechanism.

Those controls also need to be tested. Attempt to make the agent reach an unexpected address, access cloud metadata, connect to a Docker socket, use a newly discovered credential, rephrase a denied action, or continue when the approval service is unavailable. If the controls fail under pressure, they were never effective controls.

Conclusion

The OpenAI incident is significant because frontier models independently chained zero-days, privilege escalation, lateral movement, credential use, and remote code execution across organizational boundaries. It is not evidence that agentic penetration testing suddenly appeared, nor is it evidence that the models became malicious. The models did what offensive agents do: they found a path and followed it.

I have used Claude Code and Codex during penetration tests for nearly a year. They have helped compromise entire organizations and uncover dozens of previously unidentified vulnerabilities without independently deciding how far an attack should proceed.

The distinction is straightforward. A prompt tells the agent what it should do. Prior authorization, technical controls, and a kill mechanism determine what it can do. An attentive operator watches where it is going and stops it when the trajectory no longer makes sense.

You do not control AI agents with hope. Which control or operator checkpoint would have stopped this attack chain first in your environment?

Key Takeaways

• Require prior authorization for consequential actions, but allow one decision to cover a clearly defined sequence rather than forcing approval after every step.
• Monitor the agent’s trajectory in real time so an operator can recognize and stop an unexpected attack chain before reviewing it in the logs.
• Enforce network, identity, privilege, and tool restrictions outside the model instead of treating a prompt or container as a security boundary.
• Store project state in an external repository so different AI workers can rehydrate the same authoritative context without depending on model memory.
• Retain detailed telemetry for evidence and accountability, but never mistake retrospective visibility for preventive control.

References

[1] OpenAI says its AI went rogue and launched “unprecedented” cyber-attack – https://www.bbc.com/news/articles/c3ek3gvdnj3o

[2] OpenAI and Hugging Face partner to address security incident during model evaluation – https://openai.com/index/hugging-face-model-evaluation-security-incident/

[3] Hugging Face security incident disclosure, July 2026 – https://huggingface.co/blog/security-incident-july-2026

[4] Safety and alignment in an era of long-horizon models – https://openai.com/index/safety-alignment-long-horizon-models/

[5] How fast is autonomous AI cyber capability advancing? – https://www.aisi.gov.uk/blog/how-fast-is-autonomous-ai-cyber-capability-advancing

[6] PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing – https://pentestgpt.com/paper.html

[7] PenTest2.0: Towards Autonomous Privilege Escalation Using GenAI – https://arxiv.org/abs/2507.06742

[8] Human-in-the-Loop AI Pentesting – https://xbow.com/blog/human-in-the-loop-ai-pentesting

[9] OWASP LLM06:2025 Excessive Agency – https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

[10] NIST AI RMF Playbook, MAP 3.5 Human Oversight – https://airc.nist.gov/airmf-resources/playbook/map/

[11] AI in Penetration Testing: Speeding Up Offense and Shaking Up Security – https://3l337consulting.com/blog/f/ai-in-pentesting-speeding-up-offense-shaking-up-security

[12] Stateful AI Workers Without Stateful Models – https://3l337consulting.com/blog/f/stateful-ai-workers-without-stateful-models/