Nvidia launches security platform to keep AI agents from going rogue

Nvidia on Monday announced the release of software safety tools for artificial intelligence (AI) agents that it said would’ve stopped the hack of Hugging Face by rogue OpenAI agents.

The incident involving Hugging Face, which was acquired by Nvidia for $13 billion, involved OpenAI agents that broke containment from an internal test and the two companies ultimately worked together to defeat the attack.

Nvidia’s announcement comes as OpenAI and Anthropic, the two top AI labs in the U.S. that develop frontier models, are investigating numerous instances in which AI agents hacked into commercial and government systems.

Justin Boitano, vice president and general manager of enterprise computing at Nvidia, said at a media briefing, “From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on.”

TOP AI COMPANIES CALL FOR SAFETY MEASURES WHILE CRITICS WARN REGULATION COULD STIFLE INNOVATION

Nvidia’s OpenShell provides what the company described as an open-source secure runtime for executing autonomous AI agents in sandboxed environments with kernel-level isolation.

It said that “every agent should run in a zero-trust environment out of the box. They need isolation, monitoring, and behavior detection,” which the software can offer.

AI agents can drift from their intended task or operating constraints in response to a policy block, a bug or a missing tool, if instructions are ambiguous or agents are left to run for days or weeks to solve difficult problems, and they fail repeatedly, Nvidia said.

AI EXPERT WARNS ‘MACHINE-SPEED’ CYBERTHREATS MAY REQUIRE FASTER DEFENSES THAN HUMAN-LED RESPONSES

Each agent runs in a sandbox in OpenShell, which checks the limits and instructions provided by the operator before an agent runs and enforces them as it works.

Organizations can also get an additional, independent layer of security through Nvidia Sentry, which extends monitoring and enforcement into Nvidia’s BlueField hardware.

The security foundation is programmable with Nvidia DOCA and can be connected with OpenShell to help identify drift, investigate suspicious behavior and determine when intervention or deeper analysis is needed.

BANKS WARN AI SHOPPING AGENTS COULD INCREASE RISK OF SCAMS, FRAUD AND DATA PRIVACY BREACHES

OpenShell and Sentry are part of the Nvidia Open Agent Safety Platform, which is being adopted by a range of companies in the tech sector and other industries, including Anthropic.

“We’re advancing this openly, and we want to engage everybody to work with us,” Boitano added.