The OpenAI/Huggingface incident; how we should manage the imminent arrival of autonomous hacking too cheap to meter

The OpenAI / Huggingface hack, where an unguardrailed, unreleased model at OpenAI broke out of its sandbox, moved laterally within OpenAI’s infrastructure, and hacked into servers at Huggingface, will be looked back upon as the canary dying in the coalmine.

We’re going to see real adversaries doing this in earnest and exacting real damages this year, probably following an exponential. We’ll be in the shallow part of that exponential this year, but with more attention grabbing incidents that our extended families will ask us about.

In 2027 and beyond, we’ll see substantial damages that start to show up as more and more attacker constituencies adopt frontier AI and defenders, en masse, get jolted out of complacency.

We’re thus now in the ‘punctuated’ part of punctuated equilibrium of cybersecurity; the steady state we’d been in is broken, and we can expect AI security to continue to be front page news for a long time.

There are two parts to how we respond. The first is a skill issue, the second is a politics issue.

Skill-wise, there are policymakers, professional advocates, and non-cyber domain expert AI safety folks who just don’t understand cybersecurity or technology diffusion and who are in a position where they can shape policy based on their misconceptions.

This camp thinks that the main strategy to be deployed, now, is to precisely measure just how cyber-capable American models are, restrict access to capabilities deemed too dangerous in American models, and suppress open weights AI models that also have these capabilities, when:

a) attackers have free access to frontier open weights AI,

b) history shows that today’s frontier capabilities are tomorrow’s commoditized capabilities to which attackers and everyone else will have access,

c) AI cyber capabilities are the exact capabilities that need to be broadly distributed and correctly operationalized to inoculate ourselves against AI cyberattacks,

and d) open source has been the lifeblood of defensive security innovation for the past 25 years and enabling this ecosystem is key to our civilization’s cyber resilience.

Here’s what we should actually do:

• Build a government strategy around accelerating the diffusion of defensive AI cybersecurity capabilities across our government economy, society, critical infrastructure defense, defense industrial base, etc, so defenders can find and fix issues in their code and infrastructure before attackers do, and then do this in a continuous motion that operates inside attackers’ OODA loops and not vice-versa.
• To accomplish this, take a light touch around restriction and a heavy hand around adoption. Labs and open weights inference providers should be responsible for securing their APIs and knowing their customers and kicking attackers off their platforms; they should accountable for and supported in this. But it’s more important that they make capabilities available to defenders faster given that attackers will move to dark inference providers.
• Bias regulation towards forcing and incentivizing AI adoption. There are some institutions — e.g. healthcare providers, defense companies, government agencies — that should be required to adopt AI cyber defense on a timeline given the coming storm. They should be supported in this by government but also required to do this through reasonable regulations. The devil’s in the details; getting this policy right is a skill and talent question; the government and policy community need skill, domain expertise, and talent.

There’s also a politics to all of this that can’t be ignored. Any path AI takes will have winners and losers talking their book to regulators and politicians now. The labs have gone from small clusters of AI nerds debating safety in Twitter to trillion-dollar scale juggernauts managing hundreds of billions of capital. We should expect them — like any capitalist enterprise — to shape their safety narratives along the lines of their own interests.

I see the AI labs as the jewel of the American economy; but the public’s self-interest is not perfectly overlapping with theirs. We can expect them to pursue regulatory capture, anti-open-source policies, protectionism, and an interpenetration with the American state to protect their monopoly status.

In contrast, we should be asking for access to societally and economically beneficial intelligence offered at a fair price, optimally improved by a fair market, with fair regulatory guardrails, guardrailed by democratically controlled institutions.

We should be asking for and helping to build robust government safety organizations with true independence from the labs, serving as safety observatories and granted the access they need to understand AI damages and guide societal-level responses to them. Farmers not foxes guarding hen houses but where the hen houses are becoming the most important object in our civilization.

Discussion about this post

Ready for more?