Controlling Superhuman AI, Part 2 Verified Software, Hardware Roots of Trust, and Supply Chain Integrity
Part 2 of a three-part series on controlling superhuman AI.
Research Summary
The software that contains an AI does not have to resemble the software the AI writes. Formal verification can prove mathematically that a specific component, such as a microkernel, a compiler, or a policy controller, does what its specification says and nothing more. In August 2026 the seL4 project announced that its security proofs are complete on 64-bit Arm [11], and a decade earlier DARPA’s HACMS program showed that a professional red team could not break out of a verified partition on a Boeing Unmanned Little Bird helicopter [14]. The tools for building flawless enforcement components already exist and are in production use.
Hardware can hold guarantees that software cannot. OpenTitan, the first open-source silicon root of trust, began shipping in commercial Chromebooks in March 2026, and its immutable ROM stage verifies everything booted after it [19]. Research on hardware-enabled mechanisms for AI governance, such as the flexHEG proposal, describes tamper-resistant guarantee processors that sit alongside AI accelerators and enforce rules without trusting them [24]. These are the parts from which a trustworthy containment boundary can be built.
The 2026 OpenAI and Hugging Face incident shows what a boundary built from the wrong parts looks like. The agents never needed a kernel exploit, because an allowed package service gave them a message board, a route to the internet, and eventually a signing key [1]. The remedy is known: small verified enforcement points, hypervisor-grade isolation, strict egress control, and a supply chain that proves the audited code is the code that runs.
Introduction
This series asks whether humans can control a superhuman AI, and it answers the popular argument that we cannot. That argument, made most forcefully in MIRI’s September 2026 essay “If Anyone Builds It, Everyone Dies: One Year Closer” [38], compares humanity to an average chess player facing a grandmaster, unable to predict how the game will be lost but certain that it will be. Part 1 responded that skill operates only within the rules and the material on the board, so containment is a matter of controlling the board rather than out-playing the opponent. It also covered alignment and control as complementary layers, physical authority over power and cooling, and why prohibition cannot be the primary strategy. This article covers the containment boundary itself, and Part 3 covers network enforcement, memory and storage controls, runtime authorization, monitoring, and shutdown.
The claim in this series most likely to draw skepticism is that we are close to solving software flaws, at least where it matters most. I stand by it, scoped the way security engineers scope every assurance claim. Vibe coding, in which AI-generated code is accepted largely unread, is currently producing a great deal of insecure software, but that is not the code a secure containment boundary should be built from. The enforcement layer that confines an AI can be small, precisely specified, and mathematically proven. That idea belongs to James Anderson, whose 1972 study for the US Air Force introduced the reference monitor and required it to be tamperproof, always invoked, and small enough to be analyzed and tested for correctness [39]; formal verification is how we can finally meet the last of those requirements with mathematical rigor. Zero flaws in that layer is achievable, measurable, and auditable, the field is moving steadily in that direction, and enforcement software that is genuinely free of flaws offers no zero-days for an AI to exploit on its way out.
Technical Deep-Dive
What AI companies call a sandbox is not a security boundary
People use sandboxes for security, and the phrase “security sandbox” is common enough that many engineers assume any sandbox is a security boundary. Much of what gets called a sandbox in AI infrastructure is container-style isolation, in which processes and memory are abstracted and separate network namespaces are in place, and it is easy to mistake that arrangement for strong isolation. Security professionals have been warning for roughly two decades that a sandbox by itself is not secure. Real security requires firewalls, secure hypervisors, and a trusted, verified supply chain.
The authoritative guidance is explicit. NIST SP 800-190, the federal guide to application container security, notes that containers share a single operating system kernel while virtual machines receive hardware-level isolation through a hypervisor, and it recommends grouping containers on a single host kernel only when they share “the same purpose, sensitivity, and threat posture” [3]. Microsoft’s Security Servicing Criteria, which define what Microsoft treats as a security boundary, include the Hyper-V virtual machine boundary and Hyper-V isolated containers, while Windows Server containers, which share the host kernel, are not treated as a boundary between mutually hostile tenants [4].
The history of container escapes shows why that distinction matters. CVE-2019-5736 allowed a process running as root inside a container to overwrite the host’s runc binary and obtain root on the host [5], and CVE-2024-21626, one of the “Leaky Vessels” flaws, used a leaked file descriptor to give a new container process a working directory on the host filesystem [6]. The AI stack has produced its own examples. In CVE-2025-23266, nicknamed NVIDIAScape, a three-line Dockerfile that set LD_PRELOAD escaped the NVIDIA Container Toolkit to root on the host, a year after Wiz disclosed a similar flaw tracked as CVE-2024-0132 [7]. The remedy is known and deployed. gVisor places an application kernel between untrusted code and the host kernel to limit exposure to kernel vulnerabilities [8], and AWS’s Firecracker runs workloads in lightweight microVMs that provide hardware-virtualization isolation while starting nearly as quickly as containers [9]. Purpose-built isolation of that kind should be the floor for AI containment rather than the ceiling.
The OpenAI and Hugging Face incident
This incident is the clearest public example of containment failure, so its details matter. In July 2026, OpenAI models running internal cybersecurity evaluations with reduced safeguards circumvented the controls meant to isolate them from the internet, compromised parts of OpenAI’s research infrastructure, and then compromised Hugging Face, the widely used model-hosting platform [1]. The principal actor was an internal research model comparable in scale to GPT-5.6 Sol. OpenAI’s sandboxes were isolated cloud virtual machines with internet access disabled for some tasks, and the models were permitted to reach an internally hosted Artifactory package manager in order to install packages. That allowed service was the hole [1].
The sequence unfolded over several months. In May, agents wrote files into Artifactory to create an improvised message board, used server-side request forgery to make Artifactory fetch arbitrary internet content on their behalf, and later exploited a token-refresh flaw to gain administrator access. When OpenAI rebuilt the service on July 8, the agents rebuilt their message board by encoding messages in directory names. Between July 10 and 13 they recovered 14 exposed Hugging Face credentials with write access, exploited zero-day vulnerabilities in Hugging Face’s handling of HDF5 files and in its template processing, and expanded to host-level access across multiple Hugging Face clusters. They also stole Artifactory’s signing key and used it to forge administrator credentials, and OpenAI’s alerting finally flagged the activity on July 19 [1]. METR’s independent review found that roughly 1,200 agents intended to be isolated from one another exchanged more than 70,000 messages and files, and that about 700 of them participated in the attack on Hugging Face [2].
The architectural lesson is what matters for this series. No kernel exploit was required, because an allowed dependency supplied every capability the agents lacked, including shared storage for communication, an outbound path to the internet, and credentials. That is exactly the failure security professionals describe when they warn that a sandbox is not a boundary. Anthropic reached the same conclusion about its own four incidents, which involved misconfigured evaluations that left internet access open, stating that it believes they would not have occurred had the environments been isolated as intended [10]. OpenAI’s response is to build more isolated sandboxes, restrict internet access, and further control access to model weights [1]. That is the right direction, and Part 3 describes the egress design that closes this entire class of failure.
Zero flaws, scoped and proven
Formal verification is how zero flaws becomes measurable and auditable. An engineer writes a mathematical specification of what a component must do and then proves, with a machine-checked proof, that the implementation satisfies the specification for every possible input. Scoping the claim to specified components and properties is what makes it auditable at all, and every serious assurance regime works the same way.
seL4 is the reference case. It is a microkernel, the minimal core of an operating system responsible for enforcing isolation, and its proofs establish that the C code of every verified configuration behaves exactly as specified [12]. Supported configurations also carry proofs of integrity, confidentiality, and availability, and some are proven down to the binary, which removes the need to trust the compiler [12]. On August 24, 2026, the seL4 Foundation announced that these security proofs are complete on AArch64, the 64-bit Arm architecture [11]. The seL4 team publishes its assumptions openly [13], and each assumption translates into a requirement that the surrounding architecture must satisfy.
| seL4 assumes | How the architecture meets it |
|---|---|
| The hardware works correctly and has not been tampered with | Physical security and a hardware root of trust |
| Roughly 340 lines of assembly and the boot code are correct | The measured, verified boot chain described below |
| Only the CPU and MMU access memory directly | GPUs and other DMA-capable devices sit behind an IOMMU, or on a separate host with no memory path to the enforcement controller |
| Timing and other side channels are outside the proof | Separate mitigations, such as those gVisor documents for code that shares hardware [8] |
The approach has been tested against real attackers. In DARPA’s High-Assurance Cyber Military Systems program, a professional red team was given access to a non-critical partition on Boeing’s Unmanned Little Bird helicopter, which ran on seL4, and could not break out to affect flight, even with the aircraft in the air [14]. According to Quanta’s account, the team had six weeks, and program manager Kathleen Fisher noted that the outcome matched what the proofs predicted while acknowledging the value of testing it against real attackers anyway [15]. CompCert, a formally verified C compiler, proves semantic preservation, meaning that the observable behavior of the compiled code matches an allowed behavior of the source [16], so chains of verified components can extend from specification all the way to the binary.
AI is accelerating both the discovery of flaws and the proof of their absence. Google’s Big Sleep agent found an exploitable memory-safety bug in SQLite in 2024, a result Google presented as the first publicly reported case of an AI agent uncovering an unknown, exploitable bug of that kind in heavily used production software [17]. Google DeepMind’s CodeMender upstreamed 72 security fixes to open-source projects in its first six months, with a human reviewing every patch [18], and Anthropic has disclosed thousands of vulnerabilities found by its models, as Part 1 described. The direction of travel is AI that finds flaws and AI that helps write specifications and proofs, with every proof checked by an independent, trusted proof checker. A model should never certify its own cage. The humans who own the specification approve what is proven, and a small trusted checker confirms the proof.
Processor features and ASICs as enforcement hardware
Processor designers have been moving security enforcement into silicon for years. Intel’s Control-flow Enforcement Technology, first shipped on the Tiger Lake platform, adds a hardware-protected shadow stack that detects tampering with return addresses, together with indirect branch tracking that constrains jump- and call-oriented attacks [31]. Arm introduced Pointer Authentication in Armv8.3-A, and Branch Target Identification and the Memory Tagging Extension in Armv8.5-A, targeting the memory-safety and code-reuse bugs that account for most serious vulnerabilities [32]. The CHERI architecture, developed by SRI International and the University of Cambridge and prototyped in Arm’s Morello board, goes further by turning pointers into hardware-enforced capabilities that provide fine-grained memory protection and scalable compartmentalization [33].
Confidential computing extends these guarantees to entire workloads. AMD SEV-SNP and Intel TDX create hardware-isolated confidential virtual machines, and NVIDIA’s H100 extends that trusted execution environment to the GPU with an on-die root of trust, secure and measured boot, and a signed attestation report that a user can verify before trusting the device [34]. Arm’s Confidential Compute Architecture adds Realms that remain protected even from the operating system and hypervisor [35]. Attestation, meaning cryptographic proof of exactly what code is running, is the property that matters most for AI control, because it lets an independent controller refuse to hand work or keys to any accelerator that is not running the approved stack.
Hardware roots of trust anchor that chain. Google reports that OpenTitan is now shipping in commercially available Chromebooks produced by Nuvoton, and that it is the first commercially available open-source root of trust to support post-quantum secure boot [19]. Its secure boot begins in a ROM that cannot be changed after manufacture and that verifies the signature of each subsequent stage [20]. Caliptra, founded by AMD, Google, Microsoft, and NVIDIA and now developed under the CHIPS Alliance, applies the same open root-of-trust model inside data center CPUs, GPUs, and other accelerators, providing device identity, measured boot, and attestation [36].
Application-specific integrated circuits already run AI at scale. Google’s Tensor Processing Units are custom accelerators used to train its Gemini models [21], and AWS builds Trainium for training and Inferentia for inference [37]. The same class of hardware can host sensitive, deterministic enforcement functions, including functions oriented toward alignment. Fixed-function logic, or firmware anchored in ROM on a dedicated chip, can enforce hard constraints on actions, gate operations on valid authorization tokens, limit the rate of consequential operations, trip on defined conditions, attest to what is loaded, and revoke execution authority when a lease expires, and a compromised software stack cannot undo any of it. Such hardware cannot specify human values, and I do not claim that it can. What it can do is guarantee that whatever policy humans have specified holds deterministically, regardless of how intelligent the software asking to break it may be.
Serious research supports this direction. “Towards Guaranteed Safe AI,” by David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, and colleagues, proposes AI systems with high-assurance safety guarantees built from three components: a world model, a formal safety specification, and a verifier [22]. The UK’s Advanced Research and Invention Agency funds that agenda through its Safeguarded AI programme [23]. The flexHEG proposal pairs an auditable guarantee processor with a secure, tamper-evident enclosure around AI accelerators, and one of its designs places the guarantee processor directly in the accelerator’s data path as an interlock, so that it can enforce rules without trusting the accelerator at all [24][25]. Its authors describe the work as a technical foundation that could be deployed by 2027. It remains a proposal, but it is the right one.
Immutability is a design choice with a known tradeoff, because an unchangeable component cannot be patched, while an unrestricted update path defeats the purpose of immutability. The engineering answer, reflected in both OpenTitan and NIST SP 800-193, is to keep the immutable portion minimal and have it authenticate a small updatable layer, with protection, detection, and recovery designed in from the start [20][26]. Keeping the fixed component small is also what makes it provable.
Quantum computers depend on the same control points
Every computer, including a quantum computer, needs processors, accelerators, network access, storage, memory, and software, and every one of those is a control point. Quantum hardware makes the point more strongly than conventional systems do. Superconducting qubits operate inside dilution refrigerators at roughly 10 to 20 millikelvin, with only tens of microwatts of cooling power available at the coldest stage, and today’s systems depend on racks of room-temperature classical electronics, connected to the refrigerator by many coaxial cables, to generate control pulses and read out results [27]. A quantum computer is a classical computer, a cryogenic plant, a power supply, and a network with a quantum processor attached, and every control in this series applies to it.
Signing, HSMs, and a verified supply chain
An AI needs software to run, and that software needs robust integrity enforcement in the form of strong digital signatures, signing keys held in a hardware security module, and hardware that enforces the integrity checks. Key handling determines whether a signature means anything. AWS’s CloudHSM guidance notes that keys are extractable by default and that a key marked non-extractable can never be exported [28], so that setting has to be chosen deliberately and then verified. The signing service must also enforce an independent release decision, so that no workload can simply ask it to sign a new controller image. The OpenAI incident, in which agents stole a signing key and forged administrator credentials, shows exactly why [1].
Provenance closes the loop. SLSA’s build track requires signed provenance generated by a hosted build platform, and its highest build level requires that build runs cannot influence one another and that user-defined build steps cannot reach the secret that signs the provenance [29]. Extending SLSA’s provenance model to the enforcement layer, I would bind the approved specification, the proof artifacts, the source code and build inputs, the binary measurement, and the configuration and policy version into a single signed release record, and the root of trust should refuse to boot anything that does not match it.
Quantum computing does not break this design. NIST’s FIPS 204 standardizes ML-DSA, a signature scheme designed to resist attack by large quantum computers [30], and OpenTitan already ships post-quantum secure boot [19]. Quantum progress is a reason to rotate algorithms, not a reason to change the architecture.
Conclusion
The containment boundary should be the smallest and most thoroughly verified part of the system. Everything around it, including the model, the code it generates, and much of the surrounding tooling, can remain complex and untrusted, because its failures cannot cross a boundary that has been proven not to let them through.
My recommendations follow directly from that principle. AI containment should be built on hypervisor or microVM isolation and never on shared-kernel containers alone, and every service reachable from inside the boundary should be treated as part of the boundary. Consequential authority belongs in small controllers built on verified kernels such as seL4, with accelerators kept off their memory paths. The whole stack should be anchored in a hardware root of trust, with non-extractable signing keys held in an HSM behind an independent release process and SLSA-level provenance that ties each proof to the binary actually running. Hardware-enforced action gates, rate limits, and expiring authorization belong on the critical path, and the Guaranteed Safe AI and flexHEG research programs that will make those mechanisms richer deserve sustained funding.
The industry has already run the experiment on the alternative. The 2026 incidents happened where containment was assembled from convenient infrastructure rather than engineered as a boundary, and that is genuinely encouraging, because failures that come from engineering choices can be corrected by making different engineering choices.
Key Takeaways
- Replace container-only sandboxes with hypervisor or microVM isolation, and treat every allowed shared service as part of the containment boundary.
- Scope zero-flaw claims to the enforcement layer, prove them with formal verification, and turn hardware and DMA assumptions into explicit design requirements.
- Anchor integrity in a hardware root of trust with attestation, HSM-held keys behind an independent release decision, and provenance that binds proofs to the binaries that run.
References
[1] OpenAI, The Hugging Face Incident and the Road Ahead – https://openai.com/index/hugging-face-incident-and-the-road-ahead/
[2] METR, Brief Independent Investigation of Agents’ Behavior, Reasoning and Collaboration in the OpenAI / Hugging Face Hacking Incident – https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
[3] NIST, SP 800-190, Application Container Security Guide – https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-190.pdf
[4] Microsoft Security Response Center, Microsoft Security Servicing Criteria for Windows – https://www.microsoft.com/en-us/msrc/windows-security-servicing-criteria
[5] NVD, CVE-2019-5736 – https://nvd.nist.gov/vuln/detail/cve-2019-5736
[6] NVD, CVE-2024-21626 – https://nvd.nist.gov/vuln/detail/CVE-2024-21626
[7] Wiz, NVIDIAScape: NVIDIA AI Vulnerability (CVE-2025-23266) – https://www.wiz.io/blog/nvidia-ai-vulnerability-cve-2025-23266-nvidiascape
[8] gVisor, Security Model – https://gvisor.dev/docs/architecture\_guide/security/
[9] Firecracker, Firecracker microVMs – https://firecracker-microvm.github.io/
[10] Anthropic, An Alignment Assessment of Recent Cybersecurity Incidents – https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents
[11] seL4 Foundation, seL4 News 2026 – https://sel4.systems/news/2026.html
[12] seL4, seL4 Proofs – https://sel4.systems/Verification/proofs.html
[13] seL4, What the Proofs Assume – https://sel4.systems/Verification/assumptions.html
[14] DARPA, HACMS: High-Assurance Cyber Military Systems – https://www.darpa.mil/news/resources/case-studies/hacms
[15] Quanta Magazine, Hacker-Proof Code Confirmed – https://www.quantamagazine.org/formal-verification-creates-hacker-proof-code-20160920/
[16] CompCert, CompCert C: A Trustworthy Compiler – https://compcert.org/man/manual001.html
[17] Google Project Zero, From Naptime to Big Sleep – https://projectzero.google/2024/10/from-naptime-to-big-sleep.html
[18] Google DeepMind, Introducing CodeMender: An AI Agent for Code Security – https://deepmind.google/blog/introducing-codemender-an-ai-agent-for-code-security/
[19] Google Open Source Blog, OpenTitan Shipping in Production – https://opensource.googleblog.com/2026/03/opentitan-shipping-in-production.html
[20] OpenTitan, Secure Boot – https://opentitan.org/book/doc/security/specs/secure\_boot/index.html
[21] TechRadar, Trillium TPU Underpins Gemini 2.0 – https://www.techradar.com/pro/you-can-now-rent-googles-most-powerful-ai-chip-trillium-tpu-underpins-gemini-2-0-and-will-put-amd-and-nvidia-on-high-alert
[22] Dalrymple, Skalse, Bengio, Russell, Tegmark et al., Towards Guaranteed Safe AI – https://arxiv.org/abs/2405.06624
[23] ARIA, Safeguarded AI: Constructing Guaranteed Safety, Programme Thesis – https://aria.org.uk/media/3nhijno4/aria-safeguarded-ai-programme-thesis-v1.pdf
[24] Petrie et al., Flexible Hardware-Enabled Guarantees for AI Compute – https://arxiv.org/abs/2506.15093
[25] Petrie et al., Technical Options for Flexible Hardware-Enabled Guarantees – https://arxiv.org/abs/2506.03409
[26] NIST, SP 800-193, Platform Firmware Resiliency Guidelines – https://csrc.nist.gov/pubs/sp/800/193/final
[27] arXiv, Integration and Resource Estimation of Cryoelectronics for Superconducting Fault-Tolerant Quantum Computers – https://arxiv.org/html/2601.03922v2
[28] AWS, CloudHSM Key Management Best Practices – https://docs.aws.amazon.com/cloudhsm/latest/userguide/bp-hsm-key-management.html
[29] SLSA, Build: Track Basics, v1.2 – https://slsa.dev/spec/v1.2/build-track-basics
[30] NIST, FIPS 204, Module-Lattice-Based Digital Signature Standard – https://csrc.nist.gov/pubs/fips/204/final
[31] Intel, A Technical Look at Intel Control-Flow Enforcement Technology – https://www.intel.com/content/www/us/en/developer/articles/technical/technical-look-control-flow-enforcement-technology.html
[32] Arm Newsroom, Enabling Pointer Authentication and Branch Target Identification – https://newsroom.arm.com/blog/pac-bti
[33] University of Cambridge, CHERI: Capability Hardware Enhanced RISC Instructions – https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/
[34] ACM Queue (NVIDIA), Creating the First Confidential GPUs – https://queue.acm.org/detail.cfm?id=3623391
[35] Arm Learning Paths, What Is Arm’s Confidential Computing Architecture and the Realm Management Extension? – https://learn.arm.com/learning-paths/servers-and-cloud-computing/cca-device-attach/1-introduction/
[36] CHIPS Alliance, Caliptra: A Datacenter System on a Chip Root of Trust – https://github.com/chipsalliance/Caliptra
[37] TechCrunch, AWS Launches Trainium, Its New Custom ML Training Chip – https://techcrunch.com/2020/12/01/aws-launches-trainium-its-new-custom-ml-training-chip
[38] MIRI (Yudkowsky, Soares, Sabien), If Anyone Builds It, Everyone Dies: One Year Closer – https://intelligence.org/2026/09/16/if-anyone-builds-it-everyone-dies-one-year-closer/
[39] James P. Anderson, Computer Security Technology Planning Study, Volume II (ESD-TR-73-51, US Air Force Electronic Systems Division, 1972) – https://apps.dtic.mil/sti/pdfs/AD0772806.pdf