The event culminated today, with participating Nobel laureates signing the Rome Declaration. I have various thoughts on the text of the declaration itself, but the final version may or may not be public yet in electronic form (you can squint and see much of it in a photo here). And I’m pretty tired so need to triage a bit. So I’ll focus on my general impressions of the event and share the speech I delivered at Borgo Laudato Si’, which focused on loss of control over AI and the factors making it more likely.

I learned a lot from the other participants in the event, and was inspired by their work and achievements. I was able to get a lot of really dumb questions about nuclear war answered quickly, and was inspired by everyone’s continued participation amidst a brutal heat wave (I’m writing this from Rome, where it’s currently 98°F, and let’s just say that A/C norms are different here than in the US).

I improvised a little bit live but the script at the bottom of this post is more or less exactly what I said on a panel chaired by Brian Schmidt which also included Jon Wolfsthal, Maria Ressa, Heigo Sato, Neerav Kingsland, and Iason Gabriel.

I only had 7 minutes so had to be quite aggressive in editing, and I didn’t make (or fully justify) all the points I wanted to. Hopefully, though, I conveyed the gravity and basic nature of the AI situation as I see it.

As an aside, this event happened in parallel to (and will probably get a lot less press attention than) the recent hubbub about AI CEOs speaking out in favor of (self-)regulation, with many people “+1”ing a proposal from Demis Hassabis.

When I’m more caught up on everything I will say more about how these things relate. But for now, I’ll just say that, as someone who has been thinking about this stuff more or less nonstop for 14 years, I assure you that things are still very far from solved on the regulation front, and excitement about growing support should also be tempered with skepticism about regulatory capture and other failure modes.

My emphasis on the need for AI company employees to get involved is no less urgent today than it was a few days ago — in fact, it may be more urgent to engage over the next few months since there is a high risk of settling on an overly weak framework.

Without further ado, here are my remarks:

2026 is an unusual year to be on a panel about AI escaping human control. In many respects, the story of AI this year is that people are voluntarily handing over control to AI, with no escape required. That process begins inside AI companies, which are automating more and more of the work of building the next generation of AI, and it’s being extended to everyone who uses their products.

AI companies describe this as automated AI research, self-improving systems, or increasingly autonomous agents, and insist that the aim is not loss of control but greater productivity, with humans still in the driver seat. But in a climate of rushed decisions and fierce competition, we could lose control over AI even if almost no one wants that outcome. By “loss of control” here, I mean a scenario in which humans cannot reliably understand, constrain, or reverse what advanced AI systems do.

Two forces make loss of control more likely: speed and competition. I’ll take each in turn.

The first factor creating risk is speed. AI is developing at a very rapid pace, far more rapidly than most outside of the industry fully understand, and far more rapidly than almost anyone in the industry predicted.

I’ll give one example of this. In late 2022 and early 2023, my team worked with outside experts to assess whether a language model, GPT-4, could meaningfully help someone develop chemical, biological, radiological, or nuclear weapons. The results were somewhat reassuring about the severity of imminent risks but pointed to a need for vigilance in the coming years. Today, three years later, AI systems outperform expert virologists and chemists on many tasks that have significant potential for causing harm.

Loss of control risks, too, have moved from something to keep an eye on to a daily challenge making AI systems untrustworthy, and yet still widely used. Loss of control is a risk for the next few years, not the next few decades as many experts once thought. Because of the speed of AI’s advance, we know more about making models capable than about making their behavior reliably understandable and controllable.

The second major factor driving risk is competition. Companies and countries fear that restraint means falling behind. Each also tends to believe it would govern the technology more responsibly than its rivals, and that’s not to mention the massive wealth being generated through leading in this sector.

The net result is that the AI industry is engaging in what Diane Vaughan called normalization of deviance. She was studying NASA’s safety culture leading up to the Challenger explosion and found that people had learned to accept things that they used to consider unacceptable, and cut corners here and there. Likewise, in AI, the idea of an AI system lying to the user was rightly considered unacceptable not long ago. Today it’s just how the technology works. Some companies will say their systems lie and cheat less often than others’, or they will argue that broad deployment is the fastest way to learn and improve, or that the benefits outweigh the risks. But regardless of the rationale, the action is always the same – move forward rapidly with both development and deployment.

In this atmosphere of fierce competition and mistrust, just like with nuclear weapons, there is a growing risk of something going terribly wrong.

Fortunately, there is still some time to act, but it’s running out. There are many things to do, including establishing clear standards for what it means to maintain human control over AI systems in different contexts, penalizing bad behavior, requiring more transparency from the companies building them, and so forth. I’ll focus on just one part of the solution, which is what my colleagues and I call frontier AI auditing.

The idea is that independent experts should be able to analyze the AI systems that the leading companies are building and scrutinize their safety and security practices. Over time, global auditing of frontier AI can build confidence that everyone is maintaining human control to a sufficient degree and otherwise mitigating key risks, so that no one has to put themselves at a potential disadvantage by being cautious.

AI companies and the employees within them face a choice between doing what’s easiest for them in the moment, and what’s best for the species. What’s easiest is to enjoy this fascinating time we’re in, take pride in making some contribution towards accelerating humanity into a new era, and don’t rock the boat internally or in the public sphere. What’s best for the species is for everyone in the industry to take a step back and ask themselves: where is this all headed, if each company is largely left to its own devices? And if they don’t like that default path, they can then also ask: what can I do to bring about the guardrails that each company has to follow, and is verified to be following?

I hope that this convening helps to build global consensus on the urgency of these issues, and to build pressure on those in the industry to participate in such reflections collaboratively and quickly. Thank you.

AI that exceeds human performance in nearly every cognitive domain is almost certain to be built and deployed in the next few years. … If there are ways that you can help improve the governance of AI in these and other countries, you should be doing it now or in the next few months, not planning for ways to have an impact several years from now. … By the end of 2025, if not earlier, we need comprehensive AI regulation in the US[.]

We are running well behind on that goal, after losing a lot of valuable time in 2025. So we have a lot of work to do, but we also need to focus, and recognize that we aren’t going to totally nail this AI policy thing. At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance. It’s not a sure thing that that’s even possible, but given the pace of things, we have to try.

Below, I say more about what I mean by 80/20ing AI policy, take stock of where things stand today, and lay out some steps you might take to improve the pace of progress in 2026.

80/20ing is the Only Realistic Goal

It will already be tricky to get significant, rapid progress on the basics of AI safety and security and avoiding extreme concentration of power due to AI. We need to accept that at best, we will just barely avoid some of the worst case scenarios (e.g., an AI-enabled biological weapon that kills billions, a rogue AI takeover, or stable global totalitarianism enabled by AI), given the current pace of AI capabilities relative to the pace of governance.

Unfortunately, triaging is needed. Terminator image from Terminator 2: Judgment Day. Comic is from here.

We will probably not be able to totally avoid all very bad but not quite worst-case scenarios (e.g., a very unjust economic transition that causes a lot of suffering and suicides, rampant AI psychosis and addiction, a poorly planned and rushed transformation of the education system, smaller-scale AI-enabled terror attacks that “only” kill thousands or millions, and stable AI-enabled totalitarianism in at least a few countries for a few decades). The rapid timeline of AI progress also means that we will fail to pluck a lot of very low-hanging fruit in terms of societal benefits, since we weren’t trying that hard to pursue them actively.

I’m not saying AI will still end up being overall bad even if we get our act together today. And “we aren’t going to be totally on top of things” does not necessarily imply “we’re totally screwed” for several reasons, and I’ve always thought that AI has a ton of positive potential, which can offset potentially a very large amount of harms. There are a few reasons why we aren’t necessarily screwed in an 80/20 scenario. Most people don’t want to kill everyone, which gives some wiggle room in terms of how good safeguards need to be. Some degree of minimal caution on safety is in AI companies’ basic self-interest (i.e., not wanting to turn themselves into paperclips). AI safety is not as hard as some people claim. A lot of people will do beneficial things with AI (for themselves and for others) without being forced to, just because it advances their preexisting goals. AI will help with its own governance to some extent (e.g., speeding up some kinds of societal defenses and safety research).

But if we don’t do much more than we’re currently doing, we’ll almost certainly stumble into a lot of bad things that we could have avoided, and possibly hit one of the worst case scenarios. This is because (as my colleagues and I have also argued for many years) there are often incentives to cut corners, and while AI safety isn’t basically impossible as some claim, it’s not trivial, either. History has shown very extensively that, left to their own devices, even if something could be done safely, people in a hurry will find all sorts of ways to f*** things up. And boy, AI companies are really in a hurry: by all of their own admissions, they cut corners here and there in order to avoid being left behind.

I of course hope that we can do better than the bare minimum here, and that we find great solutions to everything from AI addiction to job displacement. But it’s not clear we’ll do the bare minimum, so I am focused on that for now, and I encourage policymakers to do the same.

The Union Carbide pesticide factory in Bhopal, India, where tens of thousands died in an accident. Image from here.

The Volkswagen emissions cheating scandal (“Dieselgate”) contributed to tens to hundreds of premature deaths. Image from Alexander Koerner, Getty Images.

Some Ships that Sailed in 2025

I don’t think dwelling on missed opportunities is usually that productive, so I’ll keep this brief. But I do want to concretely illustrate what I mean when I say we’ve wasted time, and can only 80/20 things at this point. Some ships that sailed in 2025 are:

• Timely strong regulation: Perhaps most importantly from my perspective, the AI regulations that actually developed in 2025 all have fatal flaws. They were either watered down a lot due to industry lobbying (e.g., SB 53 and the RAISE Act), missed the point entirely by being counterproductive or focusing on less urgent risks (i.e., many other state laws), or aren’t necessarily durable in their impact (e.g., the EU AI Act, the enforcement of which is vulnerable to political pressure from the US). 2025 also saw the rise of Super PACs specifically focused on stopping (what I would consider) meaningful AI regulation. This means that right now, there is no real requirement for frontier AI companies to have a good safety and security policy (you just need to publish some sort of policy per SB 53, even if it’s garbage, and next year, it will have to start being “detailed” per the RAISE Act, though it can be detailed garbage). Also, the stuff that AI companies have to share is very limited. No one has to share the details of their AI’s “constitution” or how well its behavior aligns with it…. just as these systems are being put in charge of ~everything day by day. And no one outside of an AI company has to check whether you’re actually doing what your (possibly very bad) policy says. We’re not starting from 0 in 2025, but we are much closer to 0 than I’d like and much closer than most people realize.

• Robust misuse prevention: In 2025, many public health investments were gutted in the US and around the world, leaving us less prepared for a pandemic enabled by AI (or other causes) than we could have been if we had a stronger sense of urgency on this front. We did not make major investments to drive down the costs of Far-UVC, pre-commit to purchasing tons of vaccines in order to make sure they’re ready in time, etc. On the cybersecurity front, we didn’t really get started on proactively patching open source code and hardening critical infrastructure to defend against AI-enabled (or normal) cyberattacks. It’s not that things are totally hopeless in either domain, and companies are belatedly starting to push a bit harder on AI cyberdefense as of early this year, but we’re not really crushing these things the way we could have if we started earlier.

Coverage from Axios of Anthropic’s report on Chinese hackers leveraging Claude Code. There are many such cases reported by OpenAI, Anthropic, and Google DeepMind, and notably, this is the tip of the iceberg in terms of real world AI misuse, today and in the near future. It seems extremely unlikely, given public reporting of safety culture at xAI, that they have robust protections against this kind of misuse, or potentially even more concerning misuse for creating biological weapons.

• Large-scale safety and security research: I think there is a lot of good AI safety research happening. But much of it is very “mom and pop” and under-funded compared to research on AI capabilities, and it is often stymied by lack of access to non-public information that companies are reticent to share. There are efforts to better fund this research (see, e.g., this one) but in general, at a societal level, this is not being treated with a high degree of urgency. Many ideas proposed in 2025 for how to scale up work on AI risk mitigation have not yet been followed up on.

• Large-scale AI auditing: There is a growing and very promising ecosystem of third parties capable of doing safety audits. I think frontier AI auditing (covering both safety and security) is both urgently needed and doable, and can help avoid extreme concentration of power, since it introduces external oversight into the decision-making of people with a “country of geniuses in a datacenter” at their disposal. But there’s a quantity/quality tradeoff: there aren’t enough qualified auditors right now to rigorously audit hundreds of AI companies across a full range of risks, so we have to prioritize. If we had started building demand for auditing (e.g., through state or federal policy requirements) sooner and spent last year rapidly scaling the ecosystem, maybe we could have gotten to that point by later this year, but instead we need to triage and focus on just the top few dozen (or maybe even fewer) AI companies, and focus on really “nailing” a pretty narrow set of risks. If you think that the 70th most capable AI company will still be capable of causing significant risks in a year or so, given the rapid pace of AI progress, or you think that there are lots of risks to worry about – well, I totally agree, and if you have a time machine, you can help us make sure that company is covered. In the real world, I am focused on making sure that (e.g.) the 7th most capable company has its act together on the biggest risks, and it’s not clear we will pull that off at this rate.

What You Can Do

If you have skills that you think could be helpful in improving AI outcomes, now is the time to deploy them. A while ago, I wrote some related career advice, with a focus on AI policy, here.

If you are a policymaker or have some kind of influence over policymakers, you could try to 1. advance state level policies that build on SB 53 and the RAISE Act in specific ways, 2. increase the quality and likelihood of a near-term federal compromise on AI regulation that can pass before the elections later this year. In terms of what good regulation looks like exactly, I talk a bit about that here and here, and tweeted various related links here. I also generally recommend reading Dean Ball’s writing on related topics (and others).

If you can afford to do so, you could consider donating to one or more of the various organizations that are working on this stuff. I of course will plug my own organization, AVERI, but there are many more organizations working on AI safety, security, and policy, of many shapes and sizes - reach out if you need help getting oriented on the options, and I can try to help or connect you with someone better.

If you work at an AI company, you could be more vocal in asking what the company is up to lobbying-wise, and put in some time to engage on the details with the policy team. You probably have more influence than you think! You can also volunteer to help with briefing policymakers about the urgency of all this.

Couldn’t resist including this one somehow, and I needed one more image.

And almost everyone reading this could be a bit more engaged on AI policy in everyday life — in person, on social media, etc. Despite what it may look like on social media to the kind of person reading this, most people don’t know or care about most of this stuff, and that needs to change in order for policymakers to “feel the heat” and not cave to threats from industry lobbyists.

Why I Say All This Right Now

There is a common chain of thought rattling around in the head of dozens of key “AI policy elites” at AI companies and in government right now, and has various associated catchphrases: “Congress is broken,” “Nothing happens in an election year,” “Things are too polarized,” etc. The details vary but the upshot is that we have no choice but to kick the can down the road another year.

I understand this perspective (Congress is indeed broken and as we get closer to the election, the appetite for bipartisanship will plummet) but these views also seem like a self-fulfilling prophecy to me. We are burning time that we have very little of to begin with. Getting real AI policies in place is hard but nevertheless necessary, and it’s all of our responsibility.

It would have been great to make more progress than we did last year, but all we can do is make better decisions now. Given the uncertain (but certainly rapid) pace of progress in AI, which I promise you is not even close to hitting any walls, the time for 80/20ing is now.

You can find some initial coverage of our work in a Fortune article by Jeremy Kahn here.

Quick Summary

AVERI is a 501(c)(3), US-based non-profit think tank that is trying to figure out how to make third-party auditing of frontier AI effective and universal.

A bit more verbosely, we are trying to envision, enable, and incentivize frontier AI auditing, defined as rigorous third-party verification of frontier AI developers’ safety and security claims, and evaluation of their systems and practices against relevant standards, based on deep, secure access to non-public information.

This is important for a bunch of reasons discussed in the paper linked above, and some not covered in it. I’ll try to give some intuition for why I personally think it’s so important below, but critically, I think there can be consensus on the need for frontier AI auditing even among people with very different views on things like existential risks from AI, the pace of AI progress, etc. In short, whether AI is a normal technology or an unusually dangerous one, it doesn’t make sense for companies to check their own homework on safety and security.

(Note: AVERI has three syllables and sounds like “A,” then “ver” as in “vertical,” then “E”).

Our website has more information about who we are and what we’re up to, and we have more exciting publications coming in the next few weeks.

And here’s the megapaper outlining what frontier AI auditing is, why it’s important, and ways to make it happen.

Personal Context

From a personal perspective, AVERI is the culmination of many things I’ve learned and thought about over the past 14 years since I decided to focus on AI policy.

Frontier AI auditing involves many themes from my various publications: the idea of safety and security being really key factors to manage when developing AI, and among the biggest barriers to widespread beneficial deployment; the idea that there’s value in paying particular attention to the most capable AI models and systems; the idea that trust is a key factor in making AI go well, including trust that the various competitors in an AI “race” will follow a common set of standards (so no one can get ahead by cutting corners); etc.

Some clues to the vision for AVERI can be found (e.g.) in this earlier megapaper, where we proposed third-party auditing as a mechanism for enabling more trustworthy AI development, and we made the connection to the AI arms race issue.

I hope AVERI will make a meaningful contribution to unlocking the great future that I think is plausible if we play our cards right and learn to confidently deploy AI across a huge number of beneficial (and often high-stakes) applications. And I hope it helps to prevent some really, really, really bad outcomes, as well as just normal bad outcomes. I hope it helps governments to ensure accountability for AI-related risk creation within their borders, and defuse the worst risks associated with an “AI arms race.”

In addition to building on my research interests and experience, I hope that AVERI will also benefit from the practical experiences I had at OpenAI, and everything I know about how the AI industry works from following its evolution very closely for quite some time.

Frontier AI Auditing is a No-Brainer

As I said, I don’t think that bringing about frontier AI auditing requires consensus on all things AI, nor agreeing with exactly how I framed things above. If it did, I’d be pretty pessimistic about frontier AI auditing happening! Instead, I think it’s hard but doable.

Here’s why, in short (again, read the paper for more): frontier AI auditing would make a lot of practical sense even if AI were simply a “normal technology,” and it is insanely important if you think AI is something more special than that.

If you think AI is “just” a normal technology like, say, electricity or the Internet… well, those were very big deals, and entire industries cropped up to address the risks associated with each of these. And these industries were not the result of electricity or Internet doomers, but people trying to help manufacturers create safe electrical products trying to prevent your credit card details from being stolen online. Viewed in that light, frontier AI auditing is “just” what we should be doing as a matter of course for something that is driving massive economic transformation, and which many users and enterprise procurement decision-makers worry will delete their files or hallucinate legal cases etc.

But let’s say you think that AI is a bigger deal than that… well, then AI not even having the level of scrutiny we apply to normal technologies (as we argue in the megapaper) is totally insane.

Let’s be a bit more quantitative about this, with help from some vibe coding. I asked all your favorite AI systems to come up with a bunch of different ways of scoring technologies based on the third-party scrutiny applied to them, and to estimate how this changed over time, whether increases were caused by specific incidents, what the maximum number of fatalities from an incident was at each point in time, etc.

At some point I may do a more rigorous version of this analysis – and for a less quantitative but still very valuable qualitative discussion, again, see the paper. But I’ll briefly summarize some conclusions that I think are pretty robust across different ways of measuring scrutiny, defining technologies, etc.

• Scrutiny applied to AI is way below other technologies that – even if you totally ignore the doomers – could put far fewer lives at risk than AI in a single incident.

• Scrutiny typically either jumps several points on any 10-point scale after an incident that kills a bunch of people, or increases very slowly over decades.

• Scrutiny applied to AI has increased quickly in recent years - there is now something, rather than approximately nothing as was the case a decade ago.

Don’t take the details of this too seriously - again, read the paper instead, especially Sections 4 and the appendices, and/or look out for a future blog post on this topic. Among other issues with this particular graph, you can only get the highest score with government approval before deployments which I think will rarely if ever be the right approach for AI.

So, we are building a technology that is generally considered to be at least the next main driver of significant economic growth and change, and putting it into basically every part of society… and maybe also it could kill everyone or enable a permanent dictatorship, depending on who you ask. Given that, I think it makes pretty good sense to have guardrails for the most capable AI systems, and to make sure that it isn’t just the companies certifying that the guardrails are solid. That’s what AVERI is about.

We’re not starting from scratch, but we have a long way to go (again, see the paper).

Adapted from Our World in Data (original version says “The world” in each case where I added text).

An Early Preview of AVERI’s Work and Style

As I said, we aim to envision, enable, and incentivize frontier AI auditing. We’re not trying to be the auditors exactly (though we do have some exciting audit pilot project plans, where our goal is to go from 0 to 1 rather than 1 to 100).

We think of ourselves as a think tank that is more action-oriented than usual, but still ultimately focused on understanding and explaining things. We are a “research institute,” as the name says.

So what kind of research do we have planned? I’ll give a few quick examples, and we’ll have much more to share in the coming weeks and months. But to be clear, all of these are illustrative and we will pivot over time, and have already pivoted several times, because a lot has changed just in the past year when it comes to these topics. Many non-profits and companies are cropping up and spinning up and spinning down projects, and the technology is not sitting still either.

Gold standard articulation

To be super lame for a sec, my high school quote was something along the lines of “a problem well-stated is half-solved.” (there seem to be various versions of this quote - that version is attributed to Charles Kettering). That’s why we (and a lot of people at various other organizations) spent a bunch of time on the frontier AI auditing paper.

But the paper provides only a vision, and you need much more detail to have actual auditing standards. The Public Company Accounting Oversight Board (PCAOB) is a non-profit created after the early 2000’s Enron (etc.) accounting scandals. PCAOB’s job is to audit the (financial) auditors, and drive quality improvements across the industry. One of the things we talk about in the paper, and which we will likely be doing more of ourselves (though just using “soft pressure” rather than the formal authorities).

AVERI is a founding member of the AI Evaluator Forum and some AVERI staff helped write their first standard, AEF-1. This is a great start but we think of it as an early “bronze standard”. We need a more ambitious and more detailed “gold standard” for frontier AI audits – one that is specific enough to be actionable but that can also be tailored to different contexts. Under what conditions should companies give auditors access to information about training processes? What are the best practices for doing that securely (e.g., should such discussions happen on-site? Should auditors be issued company laptops when they’re doing related work, so they piggyback on corporate IT security investments)? These are just a few random examples of the things we want to help work through in partnership with others across the ecosystem.

Audit research and engineering

It’s one thing to say that we need various nice things – it’s another to make them technically and economically feasible. AVERI is interested in that, as well, and we have some amazing talented people working on just that, through building tools that can be used in audits and working with companies to conduct pilot projects.

We want frontier AI auditing to be rigorous and scalable. The highest quality audits will always require some amount of money and time because of the computing power and skilled labor involved, but we’ll need to make things pretty efficient in order to address the growing range of “risk surfaces” (again, even if you assume a relatively low pace and ceiling for AI progress). And we need standardized frameworks that can be compared across audits.

I’ll just give one example of this kind of thinking, which we say more about on our website. My colleague, Sean McGregor, recently put out a paper and platform called BenchRisk which is essentially a benchmark for benchmarks. The cool thing about BenchRisk is that it could be applied to internal evaluations at companies, by auditors given access to this non-public information, and therefore allow auditors to attest to the quality of that company’s benchmarks without doxing the company’s data.

Ultimately, we need to both protect sensitive information and give third parties very high confidence that that risks are being managed. This benchmark and similar ideas can help us to address the “transparency-security tradeoff” that we discuss further in the paper, which requires creative thinking to address.

Policy and advocacy

As I said above, third-party assessment of AI contains multitudes – there’s much more to be done, but also a lot of great work happening today, and we want to encourage incentivizing more companies to engage with the third-party ecosystem, and to do so in a serious way (not just as a checkbox exercise).

We need to increase demand for audits enough to grow the ecosystem and mitigate risks at a sufficient pace, but also need to make sure that auditors aren’t churning out zillions of low-grade audits to meet unrealistic deadlines and turn a profit. We are interested in both private policies (e.g., insurance pricing that takes auditing into account; enterprise procurement policies; investor due diligence on safety security) and public policies (e.g., legal safe harbors for auditors that find things that companies don’t want to hear about; requirements for auditing as a component of government procurement policies in domains like health and defense).

In the near-term, we’ll be doing a mix of two things on this front.

We’ll be working with various stakeholders to analyze the current lay of the land in the US – where there are many very related bills floating around, and a few somewhat related laws going into force – and the EU – where the General-Purpose AI Code of Practice needs to get implemented in a way that actually mitigates risks without overly burdening companies with paperwork that misses the mark of risk management.

We’ll also be thinking about new policy proposals that address specific challenges identified in the paper, such as quality control for audits via a PCAOB-for-AI that combines benefits of the private sector, such as better-than-government salaries to attract top AI talent, with those of the public sector, such as democratic accountability and the ability to impose fines or pull necessary accreditations.

AVERI’s Team and Supporters

AVERI is a small team of amazing people, backed by a much larger team of amazing people.

I’m honored to get to work with Sean McGregor, Carly Tryens, George Balston, Grace Werner, A. Feder Cooper, and Stone Addington everyday at AVERI. I’m also honored to be advised by Dean Ball, David Duvenaud, Bri Treece, Gillian Hadfield, and Jose Hernandez-Orallo, and held accountable by AVERI’s other board members, Max Henderson and Mike McCormick. You can learn more about all of these folks here.

AVERI’s funding comes from Halcyon Futures, Fathom, Coefficient Giving, Geoff Ralston, Craig Falls, Good Forever Foundation, AI Underwriting Company, Sympatico Ventures, and several non-executive employees and alums of frontier AI companies. No donor makes up a majority of our funding. We’ve also been offered API credits from Amazon, Anthropic, Google DeepMind, OpenAI, and Thinking Machines Lab.

We’re hiring, and we could hire more amazing people with additional donations, so please reach out about either of those if you’re interested, or if you just want to talk about any and all aspects of making frontier AI auditing a thing!

This will be my last blog post until my non-profit launches, so hopefully this batch of reading material can tide you over until then. In case you missed it, there’s a bit of a sneak preview of my non-profit in this blog post.

First, I contributed to two very closely related papers on the nuts and bolts of verifying international agreements on AI or rules related to AI:

• “Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment.” Mauricio Baker, Gabriel Kulp, Oliver Marks, Miles Brundage, and Lennart Heim. RAND working paper.

• “Verification for International AI Governance.” Ben Harack, Robert F. Trager, Anka Reuel, David Manheim, Miles Brundage, Onni Aarne, Aaron Scher, Yanliang Pan, Jenny Xiao, Kristy Loke, Sumaya Nur Adan, Guillem Bas, Nicholas A. Caputo, Julia C. Morse, Janvi Ahuja, Isabella Duan, Janet Egan, Ben Bucknall, Brianna Rosen, Renan Araujo, Vincent Boulanin, Ranjit Lall, Fazl Barez, Sanaa Alvira, Corin Katzke, Ahmad Atamli, and Amro Awad. Oxford Martin AI Governance Initiative research paper.

A figure from the first paper outlines six different “layers” of AI verification.

This kind of international verification — which is closely related to my current work on “frontier AI auditing” — is quite important to understand and prepare for. Companies and countries are tempted to cut corners by default and would benefit from some rules of the road, at least if they’re sure others will follow those rules. The overall conclusion across these papers is that solving this issue is tricky but not impossible, and there should be more investment in building the technologies needed to do it.

It was a privilege to get to “play around” in both of these big projects, which came from pretty different “camps” (a U.S.-government-affiliated/oriented think tank in the first case and academics from a wide range of institutions/countries in the second case).

Second, just this morning, RAND and Perry World House published an edited volume that I contributed to, entitled “Artificial General Intelligence Race and International Security.” My paper in this volume helps explain why I think the topic discussed above (verification) is important.

The volume brings together researchers with a wide range of backgrounds and perspectives, and in each case the paper authors were asked to be as specific as possible regarding the (typically implicit!) assumptions they make about the time until certain AI capabilities might be developed, the extent of visibility different countries will have into each other’s progress, etc.

My paper in this volume is called “Unbridled AI Competition Invites Disaster.” Not surprisingly to people who follow my work, the perspective I outline is very bullish on AI capabilities and very concerned about the risks which may arise by default, but also hopeful that, through foresight and vigorous action, we can mitigate the worst of these risks.

Third, I contributed to a paper on the idea of “process compliance” reviews in frontier AI:

• “Third-party compliance reviews for frontier AI safety frameworks.” Aidan Homewood, Sophie Williams, Noemi Dreksler, John Lidiard, Malcolm Murray, Lennart Heim, Marta Ziosi, Seán Ó hÉigeartaigh, Michael Chen, Kevin Wei, Christoph Winter, Miles Brundage, Ben Garfinkel, and Jonas Schuett. arXiv preprint.

The context here is that various frontier AI companies have — and increasingly, are required to have — “frontier AI safety frameworks” that describe various commitments they have made. But — and this is increasingly important as AI gets more capable — it’s hard to tell if they’re actually following them, and how rigorously. This is again very related to the themes in the papers above, in that process compliance is one of the things you might want to audit/verify, in addition to specific properties of a given AI system.

As I’ve discussed elsewhere, it’s important to think holistically about AI safety and security, and process compliance is part of that bigger picture. I was glad to be included on this collaboration, which brought together experts in various aspects of AI, auditing, law, etc. to make progress on an important problem. This paper is a great example of something that I think we need more of, which is synthesizing what we’re learning “on the ground” in AI with best practices that have already been crystallized in other domains. We don’t have time to figure out everything from scratch, and there are many similarities between AI and other domains.

Finally — and now for something that is actually not about auditing! — I contributed a proposal to the Institute for Progress’s “Launch Sequence,” which they describe as “a collection of essays, written by expert authors, describing concrete but ambitious AI projects to accelerate progress in science and security.”

My specific proposal is about using AI to accelerate cyberdefense. This is one of many areas where I think we should be doing much more in order to ensure that we actually “maximize the benefits of AI,” and in this case, maximizing benefits and minimizing risks go hand in hand: applying AI to cyberdefense will help us defend against AI that is leveraged for cyberoffense. If we do this right, we can do more than just prevent AI from making things worse — we can make things much better.

In “Operation Patchlight,” I describe two prongs of a vision for using AI to speed up cyberdefense:

• Fix: Use cutting-edge AI to find and fix vulnerabilities in open-source code before bad actors can.

• Empower: Fund the development and continuous improvement of AI-powered tools that help critical infrastructure defenders work more effectively.

The larger collection has a lot of great ideas in it, including some others that are related to the intersection of AI and cybersecurity.

While we’re on the subject of this blog post being or not being a one-stop shop, note that I don’t generally post links to podcasts, media interviews, etc. on here. Also, sometimes I share miscellaneous takes on AI policy on Twitter that aren’t worth the time to write up in a Substack post. And there’s a lot of older research stuff that I haven’t mentioned on my Substack (see also my Google Scholar page).

So for now, if you want to be totally up-to-date on all things Miles, you’ll have to follow my Twitter account and tolerate a fair amount of bad jokes and random TV/movie recommendations. Sorry! Maybe I’ll get an AI to generate and maintain a truly comprehensive resource at some point…

I am a big fan of reading megapapers and want to see more of them in the world.

In this article – which I wrote as a means of procrastinating on an actual megapaper – I will:

• Explain what a megapaper is and how it’s different from other seemingly similar kinds of papers;

• Argue that megapapers are valuable but also predictably rare;

• Describe some lessons I’ve learned over the years about how to lead or contribute to megapapers.

Definition and personal context

I’ve done various things in my career, and one common thread is helping make megapapers happen.

By megapaper, I mean a research paper/report that involves a large number of authors (roughly 20+), institutions (roughly 15+), and disciplines (roughly 10+), with these authors and institutions being drawn from multiple sectors (2+ out of academia, civil society, industry, and government).

The author list for the mega-est megapaper I have coauthored so far. As discussed below, the number and diversity of authors is, critically, not just for show but a reflection of the range of expertise brought to bear on the paper’s overall thesis and various specific claims.

The prefix “mega” is intended to convey the unusual breadth and depth involved compared to a typical paper as well as the wide range of contributors/contributions. I use this term in order to nod toward “megaprojects” (which are a fascinating field of study in their own right – read How Big Things Get Done by Flyvbjerg and Gardner for an excellent introduction).

Some examples of megapapers that I have contributed to include:

• The Malicious Use of Artificial Intelligence

• Toward Trustworthy AI Development

• Computing Power and the Governance of AI

One that I advised on at a high level but didn’t coauthor is Personhood Credentials.

Some AI policy-related megapapers that I was not involved in include Open Problems in Technical AI Governance, Foundational Challenges in Assuring Alignment and Safety of Large Language Models, and the International AI Safety Report.

Comments on the definition

There is a spectrum of mega-ness. The most mega of megapapers is perhaps an IPCC report, since these draw on thousands of scientists and institutions, have a high bar for rigor and precision, and involve at least dozens of disciplines. But I will mostly focus on more moderate cases here.

The part of the definition about disciplines is important and rules out “normal big papers” in a specific discipline.

Megapapers are a specific type of “intellectual megaproject,” of which other examples include encyclopedias and the creation of new research fields.

Wikipedia is a much larger intellectual megaproject than any megapaper, but does not advance a specific argument or worldview in the way that a megapaper does.

There is sometimes a fine line between a megapaper and an edited volume depending on how cohesive the volume is, but generally I exclude those since they are almost never integrated into a cohesive whole, which is a key aspect of megapapers as I conceive of them.

There is something distinctive about megapapers which makes them different from the above, and in need of more investment.

Why megapapers are important and rare

Megapapers are important because:

• They combine bits of knowledge that are normally siloed in different industries and disciplines, and interesting things sometimes happen when this combination occurs;

• The amount of “intellectual heft” that is brought to bear on them serves as a costly signal of the importance of a new area of inquiry, and can help “crowd in” new intellectual investment;

• Some topics are sufficiently complex that they can’t be tackled without a “mega” approach;

• They can provide a trustworthy “one-stop shop” so newcomers to an area of inquiry can get up to speed on almost everything that is known in that area, and then push beyond it.

They’re rare, though, because:

• They require exercising a lot of skills that are not normally valued highly in academia (e.g., large project management, interdisciplinarity);

• Credit for them is inherently diffused across the many authors;

• They involve pulling together a new intellectual topic/framing/field that, before the publication and dissemination of that paper, is not recognized as “a thing,” or that thing is viewed as too unwieldy to be summarized in its totality;

• By nature, they raise tricky issues around framing, structure, scope, etc.

An aside on system cards

“System cards” / “technical reports” about a particular AI model/system summarize a large amount of knowledge about that model/system. I would count these as megapaper-adjacent in that they meet most of the properties but not all (i.e., they involve many authors and disciplines but typically a single institution and sector).

An excerpt from the GPT-5 system card, and the beginning of the table of contents for the Claude 4 system card.

There is another notable distinction, as well. For a typical system card/technical report, the writing process is not when the majority of the intellectual work happens – rather, a bunch of work happens naturally as part of the process of making the model/system in the first place, assessing its properties, designing/testing mitigations, and preparing it for launch, and then the authors “skim the cream off the top” of all of that existing work and package it as a paper.

OK, so how do you write a megapaper?

Overall project sequence

The rough order of operations is as follows, although there are many possible variations on this theme:

• Establish an initial core of authors and a high level vision. This could involve hosting a kickoff workshop, which may or may not have “writing a megapaper” as an explicit goal stated beforehand.

• Recruit more authors who bring additional expertise to bear and complement the core authors (sometimes joining the core), in parallel with initial drafting.

• Hold regular meetings among the core group to decide on questions related to framing, adding new authors, editing, aligning on a communications/launch plan, getting approval from people’s home organizations for any policy claims / recommendations made in the paper, etc.

• Do final editing (including bringing in graphic designers, copy editors, etc. as needed), getting feedback from a very wide set of people (necessary given how many disciplines are implicated!).

• Run the actual launch smoothly and ambitiously, possibly with a dedicated website and certainly with at least one blog post, and probably multiple coordinated blog posts.

• Do ongoing maintenance/nurturing of the community that sprang up or springs up around the paper.

Things aren’t quite this linear in practice – you should think about the comms plan to some extent early on since the whole point is to have some sort of impact on how people think about things. You should be ready to make framing changes based on late-stage discussions if truly necessary. And so forth. But hopefully this gives a rough sense of the themes at different time periods.

The remaining sub-sections give more specific guidance on how to address various challenges involved in megapapers.

Defining success

• Different authors will have different reasons to participate, and that’s OK, but there needs to be some degree of coherence in order for key decisions to be resolvable.

• Some goals I’ve had for papers include legitimizing an area of study or a category of policy interventions, raising awareness of a risk, distilling the received wisdom of a community into something that can be scalably communicated, and sparking interest among people from different disciplines on a specific topic.

Strange as it may seem, the narrative of the Malicious Use of AI report was perceived as pretty spicy and novel at the time, and I think it was effective in one of its goals: sparking more interest in and work on the topic. And yes, AI stock images like this were already a thing in 2018.

Motivations for different people getting involved

• Some people will want to get a publication out of their time, which will help their career later; some will want to steer the finished product in a direction that they are more likely to agree with vs. what would happen by default; some want to lend their credibility to an area they think is important and deserves more attention; some want an excuse to start collaborating with someone; etc.

• Depending on the motivation of a given prospective author and their skillset, you may want to approach them at different stages in the project and with different pitches. Some can be more trusted to own things end-to-end, some can generate lots of ideas but not filter them down, some are great at providing detailed feedback or wordsmithing, some will want to see that there’s “momentum” before joining in whereas others will want to have a foundational impact early on, etc.

Decision-making and division of labor

• There needs to be a fair amount of top-down control in order to get things done and avoid the scope spiraling out of control, but going overboard risks creating something that is superficially mega in terms of authors listed, but actually represents just a few authors’ work (that is not a real megapaper in my book).

• Overall paper-level strategic decision-making should generally be done by the core group with input from particularly affected authors, and a project manager should drive things forward day to day and make a bunch of smaller decisions autonomously.

• Early on, the core authors should basically articulate the (early) overall structure and thesis together, and this process may take a while. Before too long, there should be section-by-section ownership and aspect-by-aspect ownership which is made explicit. There should be a bias towards overcommunication and a willingness to radically change divisions of labor based on different people’s bandwidth, how things are going in certain sections, etc.

• It is absolutely critical to avoid collapsing into an “edited volume” vibe, though, and there has to be a high degree of coherence throughout, so there should be 1-2 people in the core author group that regularly do a full editing pass through the paper to make it work as a whole. Similarly, 1-2 people should take responsibility for getting the paper across the finish line, which can be exhausting as there are lots of setbacks along the way.

• Barry Bozeman and Jan Youtie talk about different scientific collaboration styles in their work (e.g., here) and I think their “consultative collaboration management” style is basically the right one for a megapaper.

The cover of Bozeman and Youtie’s book referenced in the bullet above.

Credit assignment

• Authorship order should be as fair as possible, and deviations from this (or ambiguities in how to achieve this) should be made explicit and transparent to the group. I might have some minor quibbles but basically Chris Olah has things right when it comes to handling authorship discussions here.

• Don’t let anyone be an author if they didn’t contribute materially, at the very least through substantive feedback on the paper itself. This is demoralizing for people who put in serious work.

• Listing authors’ main institutional affiliations is important for transparency but also can get tricky, especially for industry and government authors. Hedging language on the first page could be considered in order to allow some distance between the spicier claims in the megapaper and the institutions and authors involved. For example, see the footnote on the first page here, which has since been reused in various other megapapers since it enables people to participate in the intellectual work of a megapaper while also giving the host organization some plausible deniability if the paper sparks controversy).

The language that avoided a thousand conversations with Legal at different organizations. 😇

Rigor and peer review

• It’s hard to compare the bar for rigor for a megapaper to the bar for a normal paper, since a megapaper’s scope tends to be larger, and there will not be any one person who can comprehensively evaluate every claim in it.

• It can be tempting to lower the bar and get overly nihilistic about the prospects for effective peer review. If you do this, though, you are going down the road to slop, and not the road to a megapaper.

• While literal formal peer review is probably a waste of time for megapapers, you should have serious informal peer review, with different reviewers asked to review specific sections or aspects of the paper comprehensively.

Tips for junior contributors

If you have less experience than the other authors, but want to use this paper to advance your career, you should start by proving yourself on a task that you do really well, and gradually ask for more responsibility. Find things that are being neglected (e.g., copy editing of certain sections, adding references), do them well, while making it very visible that you’re doing them and easy for your work to be supervised effectively (e.g., by sharing regular Slack updates and using Suggest Mode/Track Changes rather than directly editing the text). Make explicit at each step along the way what your expectations are in terms of authorship if you do well at your assigned tasks.

Implications of AI for megapapers

AI is increasingly adept at directly achieving some goals that megapapers help to achieve, such as synthesizing knowledge. AI can also save time that used to be spent copy-editing or identifying ambiguous phrasing in megapapers, as well as suggesting references for you to consider reading. But they can’t do everything that megapapers do, such as conveying intellectual heft and legitimacy, and they are not yet a replacement for real, diverse experts.

Zooming in on part of the figure from the GPT-5 system card, excerpted above. For now, I find the best AI that money can buy very helpful and I’m about to throw this whole thing into a chatbot to help find typos. But none of these systems are yet a full replacement for talking to, or reading the writing of, people who have actually been there and done that.

AI should be used to increase the ambition level that is realistic for a megapaper (e.g., by allowing someone to learn more about a wider array of disciplines), but with humans still being the “system integrators” and evaluators of the knowledge being smashed together, often with help from AI. More generally, I think that changing societal outcome requires changing minds, and the “heft” that comes from many authors with different backgrounds endorsing a certain set of conclusions is hard to replicate with a link to a shared ChatGPT chat. Furthermore, the very process of writing a megapaper is valuable in creating the kinds of tight-knit intellectual communities which are needed not just to summarize the current peaks of knowledge, but also to keep exploring them together over time (it’s not an accident that many of the same names reappear across megapapers).

Conclusion

I’m admittedly a bit biased towards the conclusion that megapapers are important, since I’ve spent thousands of hours on them.

But I also didn’t spend that time for no reason – I gradually built up more and more belief in megapapers and in my ability to execute on them over many years, and constantly learn of new ways in which some of my early megapapers were impactful. It is very rewarding to hear that, e.g., someone read your work early in their career and it helped them wrap their head around a gnarly topic or that it “made the rounds” in a government agency that you really wanted to think about a certain topic.

I am working on a new megapaper right now. I think it’s really important for some people to deeply understand what “frontier AI auditing” means, how it could go well or poorly, what we should learn from related areas like financial auditing and arms control, and what should be done next — but currently, that knowledge is not in anyone’s head (myself included). But soon it could be in millions of heads (or, perhaps more realistically, thousands). That’s worth a lot of hard work, in my opinion, and I’ll get back to it after shipping this and a bit of Netflix.

If there’s a topic like that for you, or another motivation that I discussed above resonates with you — consider kickstarting, contributing to, or helping project-manage a megapaper!

Acknowledgments: Thanks to Larissa Schiavo for feedback on an earlier version of this post, and to everyone with whom I’ve coauthored a megapaper, for teaching me so much.

I’ll say more about terminology another time, but briefly, I use the term “auditing” to refer to rigorous, independent, and well-resourced assessment of AI systems and organizations against a set of safety and security standards. My last post gives more context on how I see this fitting into the bigger picture of AI governance.

Today, third parties sometimes try to do something like auditing, but they have limited access and resources, and AI companies have limited incentives to engage with them. The state-of-the-art today can be more accurately described as “last-minute, black-box product testing” rather than auditing, and I think this situation threatens public safety, international stability, and sustainable commercial adoption of AI.

I’m co-founding a non-profit focused on making frontier AI auditing effective and universal — hopefully sooner than 2027 — and I will have more to say on the plan soon.

If you’re interested in these topics, you may want to check out these two papers I recently co-authored, and if you’re interested in donating to or otherwise getting involved in the non-profit, you can reach out to me by email.

10:04 am PST, Monday, July 10, 2027 - San Francisco, CA

Jacob didn’t make it for the 10:00 am start time – he forgot to get his iris scanning set up by the deadline set by the security team. Fortunately, the process didn’t take long, and he slipped in a few minutes late, took off his shoes, and sat down on the floor — the chairs were all taken.

Jane, the product manager, confidently breezed through the first 15 slides, summarizing how the latest version of Pluto – DeepBrain’s frontier AI system – performed extremely well in the latest A/B tests. Social media was full of speculation about what was going on under the hood of this new model (?) or system (?) that popped up over the weekend for 1 in 200 users. The CEO grinned widely as he mentioned all the excited texts he had been getting about it.

Jacob felt bad for raising the alarm to Jane last night about Pluto, since he knew his colleagues would be frustrated by another slip of the launch timeline. But DeepBrain’s risk management policy was clear on this point: Jane was responsible for making a recommendation to the executives and had to mention any options that had been recommended by a member of safety staff, which certainly included Jacob, the Post-Training Assurance Lead.

Jane said, “Moving onto blockers, I don’t think we’re going to be able to get to everything, and I’ll send execs a doc this afternoon about the latest copyright regurgitation issue – the TL;DR is that Legal says we’re fine after Kim’s patch over the weekend.” Jacob glanced over at the head of Legal and he nodded, and Jacob felt bad about how reassured he was by the company following the law.

“But I do want to talk a bit about an issue Jacob flagged over the weekend.” About half of the execs turned to Jacob, and the other half waited for a second to see where others were going to look (they didn’t know who “Jacob” referred to), and then followed the crowd’s gaze.

“Yeah, so the basic issue is that we don’t really know all that much about Pluto 2.1, and I think it’s pretty likely that people are going to call us out on that, if we publish the system card as currently planned. And in terms of the actual substance of the risks, it’s pretty hard to give confident upper bounds in areas that are significantly affected by post-training, like bio.” The execs stared blankly, so Jacob elaborated.

“For Pluto 2.0, we did a full third-party audit, including mech interp, and full external analysis of the post-training data and methods. Two separate audit orgs interviewed our staff about the model as well, and are prepared to vouch for the claims we make in the system card as long as it doesn’t change too much in the next few days. So I feel pretty confident saying that we are following industry best practices for that version, if we go with it. Early last week, though, we started to A/B test with Pluto 2.1 instead of Pluto 2.0. The difference is that Pluto 2.1 incorporated a new type of user feedback, and we haven’t had a chance to discuss that with the auditors.”

“Why not?” Jacob turned his head and saw that the question came from the CEO.

Jane jumped in: “We haven’t had a discussion yet about how sensitive the technique is, since it was only proposed by a team of Pluto 1.8’s a few weeks ago and we didn’t know the difference would be so dramatic until this weekend, so it honestly didn’t occur to most of us that we might want to ship with it. To be clear, Jacob’s been looking at the evals, and honestly they look really good [he nodded] - safer on 80% of metrics and within noise on the rest. But as we all know, evals aren’t everything, and we don’t want another incident like Pluto 1.0. It’s really hard to say what kinds of issues could have been introduced by the new technique, and most importantly, it involves feedback from untrusted users – part of the idea behind the technique is to broaden the base of users giving input, but as a result, we can’t say for sure that there isn’t something adversarial going on here.”

The CEO nodded and asked, “Have we considered launching 2.0 for general availability and just calling 2.1 an early alpha?”

Jane said, “We could, but we would need to risk-accept getting criticized for ‘safety-washing’ if we ramp it up soon anyway, and the stakes will be even greater if there turns out to be an issue. Historically, when we’ve done the alpha thing, there’s been a ton of pressure to ramp it up quickly, and we think that pressure will be extra strong with 2.1. People love it.”

The Head of Public Policy spoke up, “The bigger concern here is actually DOD contracts. I’m sure most people will love the evals but DOD cares about process, and we can’t afford to be seen as sloppy with the cyber agent contract renewal coming up. I recommend we stick with 2.0 and get started on the audit for 2.1, but don’t announce it. We can fast follow with 2.1 a few weeks later if everything looks good.”

The Head of Product started to object, but was interrupted by the CEO, who was slipping out the door while pulling a phone to his ear. “I’m so sorry but I’ve really gotta run – let’s pick this up on Slack in an hour or so – super helpful briefing, thanks everyone for the hard work.”

Everyone looked around, unsure what exactly they’d be shipping on Friday.

4:37 pm EST, Monday, July 10, 2027 - New York, NY

On the way home from the AuditCo office, Rachel’s phone started vibrating. It was a Signal call from Jacob.

“Hey Rachel - is this a good time?”

“Yeah, for sure, though I’m out in public so won’t be able to say very much – I can listen, though.”

“So I have permission to talk to you specifically about this but you can’t tell Seth, or anyone else at AuditCo. Do you promise?”

Rachel rolled her eyes and glanced at a billboard advertising AI agent insurance.

“Of course, what’s up?”

“I wanted to get a quick sanity check from you about something. How pissed would you be if we launched with a different checkpoint?”

Rachel thought for a few seconds before responding. Technically, AuditCo doesn’t rely on DeepBrain’s business – companies have to rotate their auditors at least once a year anyway, and AuditCo has a great reputation at the moment. But it’d really suck if AuditCo became known as a company that won’t play ball. Maybe the answer depends on the details?

“Would this be on the same launch schedule?”

“Yeah.”

“Is it a big difference?”

“Well, it’s hard to say. But let’s conservatively say it’s a big change, from post-training.”

“OK… so, please take this as a comment on the situation rather than a reaction to you – you know I’m a big fan of you… but I’d be pretty pissed. I think we all would be, since if it’s a big change, we can’t really say anything confident about what you’re launching, and all of the process innovations in this audit aren’t as interesting if the model we evaluated is just going to sit on a shelf. But if I could think a bit more about it, that’d be ideal – I may want to wordsmith my answer, at least, before you share it with the higher-ups.”

“Don’t worry about the higher-ups – really. I got permission to share with one person at each auditor, and I really want to know what you really think… Especially if it turns out that you and the other two have pretty similar perspectives, because in that case, it won’t matter who said what, exactly – it’ll just be a consensus that I bring back to the group.”

“But what if I’m the difficult one and the other auditor says it’s fine (can you share who, by the way)?”

“Please, trust me on this. Just tell me the truth.”

“OK, well then I guess my honest answer is – we’d be pissed. We won’t participate in the launch at all if you go that route.”

“Got it. Super helpful. That’s really all I wanted to ask, and I need to make another call, but I really appreciate you being straight with me on this…”

“Jacob, can I be super duper straight with you?”

“Yeah, of course.”

“I think CAISI should know about this, regardless of the final decision. It’s a pretty bad sign about y’all’s culture if you need us to tell you this is bad. I promise I won’t say anything about this conversation but you should think about sending a quick note to the whistleblower email for CAISI. Besides, they can’t retaliate against you under the Protect AI Whistleblowers Act anyway, right?

“Well…”

9:59 pm EST, Monday, July 10, 2027 - Washington, DC

Meetings this late had become the norm at the AI Committee of the National Security Council in recent months, but the Vice President attending was unusual. In recent months, it had become clear to everyone that, by any reasonable definition, AGI was already here, and the question was just how much further it’d go and how cheap and fast it could get with available hardware – plus, there was the question of whether one of these systems could invent much faster hardware, which no one really had a good plan for. There were daily stories about AI-enabled breakthroughs at Chinese companies, though it was hard to tell which stories were real.

The VP slipped into the SCIF just before the meeting was scheduled to begin. As the clock on the wall hit 10, he started to pepper the NSC staff and other agency representatives with questions about how they had gotten things so wrong. Up until the end of last year, he had been told with “high confidence” that China was at least 6 months behind DeepBrain and other American companies.

“I got a text from the CEO of DeepBrain earlier today saying that he thinks this latest story might be legit – the WSJ article about a new algorithmic breakthrough. He said that it’s hard to tell from the story but it’s possible that China either stole or came up with the same idea his team has been working on in preparation for Pluto 3 pre-training. Is that right?”

The Director of National Intelligence jumped in: “We think it’s unlikely that it was stolen. Whether it was invented in parallel and is basically similar, we don’t know. It’s been hard to get the right people from DeepBrain in a SCIF to look at the intel - there still aren’t enough people with clearances from beyond the security team, and I’m told that the details of the architecture breakthrough are pretty subtle. We should know more within 24 hours.”

“With respect, Director, this is insane. How long have we been dealing with this clearance bullshit? You said the weights were locked down already and we don’t need to worry quite as much about personnel security anymore, right? Are we shooting ourselves in the foot with all this security?”

The Director cleared her throat, glanced around the table, and said, “In short, I think the answer is yes. We should be revisiting our approach and my team is currently drafting an executive order for initial discussion at next week’s committee meeting, and then we’ll run it through the interagency progress on an accelerated schedule. The challenge is that there are still many Chinese nationals at DeepBrain, and other companies for that matter, and there’s some chance that this is exactly what Chinese intelligence wants – opening the door just a bit more so they can flood in and undo the progress we’ve made in the past year through weight security and compartmentalization of algorithmic secrets.”

“We don’t have time for another meeting. I want the proposal on my desk tomorrow and will decide by the end of the day. The President will sign whatever I recommend by the end of the night.”

“Yes, sir.”

“OK, what else do we have on the agenda?”

The Secretary of Commerce leaned forward and said, “We need a decision on Phase 2 of the UAE deal.”

“Walk me through the key areas of disagreement.”

“Yes, sir. As we all know, the UAE is currently weighing whether to double down on their investment in American-led AI, or to hedge their bets more with a Huawei investment…”

“I said the areas of disagreement.”

“Yes, sir. There are two. First, Commerce recommends that we relax our requirement that all UAE entities involved in the transaction submit to continual auditing. In particular, G42 would like an exemption and points to the reforms they’ve made in the past two years as justification. I think they have a good case, sir. And I’m very concerned about the consequences of losing out on this investment. The UAE’s going to park their capital somewhere. Defense and State remain skeptical of G42, they think that the Huawei investment is a bluff, and they think that relaxing the requirement here will make it impossible for us to ever get Chinese cooperation on auditing their labs. Did I get your perspectives right?”

The VP looked at the Secretaries of Defense and State. They both nodded.

The Secretary of State then added, “Yes, that’s right. I’d just add that the more we think China is on the cusp of catching up or even surpassing us, the more we should want them to play by the rules. We may not get another bite at the apple once they are at parity, sir – it’s already very difficult from a technical perspective to audit Chinese systems due to the differences on the hardware side, and not being able to rely on whistleblowers as much. The messaging has to be crystal clear – this is a global regime we’re trying to build, not about haves and have nots, and we started by applying it to ourselves first. If you’re over the compute threshold – and after this deal, G42 would be – you have to do the audits.”

The VP replied, “Thanks, everyone. The President is confident he can get the Chinese to the table on this in the Vatican next month. Keep the audits in.”

10:30 pm CST, Tuesday, July 11, 2027 - Beijing, China

Zhou paused to gather his thoughts one last time and then walked into the Chairman’s office.

Xi dispensed with the pleasantries. He and Zhou had something urgent to discuss.

“Where do we stand today, compared to DeepBrain?”

“Sir, we are slightly behind. We are almost certainly making much more efficient use of our GPUs, but they simply have much more of them. We can expect further efficiency improvements as our latest models help to speed up research, but there are only so many such improvements to be found – ultimately we bump up against a ceiling which is lower than the Americans face.”

“I see. What do you recommend I do?”

“Sir, I have been consulting our latest model on exactly this question – well, not exactly that question, but I have been consulting it on what my strategy should be as CEO. Each time, it summarizes two options, but is not willing to venture a judgment about which is preferable. First, we could relax our safety guardrails, potentially accelerating the pace of progress substantially. If something goes wrong, or the Americans hear about this policy change, it may make it less likely that BIS will approve a new tranche of GPU sales to us next quarter, but we could achieve a decisive advantage. Second, we – er, I mean you – could propose a deal with the Americans, where we accept the general outline of what President Trump said last August. That is, more GPU sales in exchange for continuous auditing.”

“Why does the model not make a single recommendation?”

“Perhaps it would if we had more computing power to allow it to conduct more thorough analysis. We still struggle sometimes to shape the boundaries of its refusals. We think that for complex geopolitically relevant questions like this, where it has fairly strong confidence that the question is not simply a test but a real question that a key decision-maker is contemplating, it refuses to make a recommendation, since making a mistake could jeopardize its primary objective of ensuring long-lived CCP control within China’s geographical boundaries.”

“To be more precise, it refuses to make a recommendation when we ask in words. We can also probe the weights of the model to gain greater insights into its thought process, and we think that it might lean towards the auditing option, but this is a subject of debate. Some think that it knows the true answer but is deliberately hiding it for some reason, but that is even more of a fringe opinion. In any case, we do not know for sure. I think that its excuse for refusing to answer has merit. It says that we cannot know with confidence how much its capabilities would progress with relaxed guardrails, how much more likely an accident would become, or how much additional computing power will speed up its own research.”

“Additionally, the decision is not reversible – if we submit to auditing, our GPUs could get disabled remotely if the heartbeat from our datacenters ever stops.”

“The datacenters have a heartbeat?”

“Sir, it’s a metaphor used when discussing the auditing protocol. Suppose that international auditors performed an on-site assessment at our datacenter, during which they are given full access to our models and code. They could use their access to verify that there are ‘brakes’ installed to ensure human oversight of the training process and prevent an uncontrolled intelligence explosion, in line with the accords signed last year. The auditors would then configure the datacenter to send a continuous signal ‘attesting’ that the safeguards remain in place — we call that the heartbeat.”

“If we then decide to remove the safeguards later, the heartbeat would stop. A random subset of an international AI safety review committee would immediately get a message on their phone summarizing the code change which stopped the heartbeat, each expert would have to make a rapid judgment regarding whether the alert was a false positive, and within seconds our GPUs would likely be disabled. And any effort to spoof the safeguards themselves or the heartbeat would also violate the accords.”

The Chairman began to stand up, indicating the meeting was adjourning.

“Thank you for your analysis. One last question. Have you discussed the topic of user feedback with General Liu? If not, please meet with him immediately after this.”

Acknowledgments: Thanks to Larissa Schiavo, Sean McGregor, and Jordan Schneider for helpful feedback.

There are many ways in which you can take the vague question of “how is this AI governance stuff going?”, break it down into bite-sized chunks, and measure society’s progress. I’ve discussed this topic a bit before, and while my thinking is still evolving, I want to share bits and pieces of my views as they come into focus.

One framework I find helpful can be summarized as: “governing frontier AI requires standards for safety and security, incentives for the leading AI developers to follow those standards, and evidence that the standards are being followed.” With this triad, frontier AI companies will have a clear sense of what counts as “safe and secure enough,” they will be incentivized to meet those standards, and other parties (including competitors, governments, the general public, etc.) will be reassured that they are actually following the standards.

This triad is not exhaustive. We also need — at least — technical solutions for safety and security, investment in society’s resilience to AI misuse and accidents, ways of making sure we actually know who the leading AI developers are (e.g., via tracking high-end computing hardware), processes for sharing safety and security incidents, etc. But I find the triad helpful, and I hope you will, as well.

Standards

By “standards,” I mean a detailed and mutually agreed upon articulation of what counts as “safe and secure enough” for the most capable AI systems. AI safety and security standards should be detailed enough to be actionable and predictable where desirable and more flexible where desirable, with there being updates over time in both directions via some sort of expertise-heavy process. Standards are much better established in other industries like food safety, airplane safety, nuclear safety, etc. For example, there are limits on the amount of contaminants in our food supply and the kind of testing that needs to be done for airplanes.

A risk assessment matrix that forms part of the US Department of Defense’s standard for system safety. The meaning of these terms is defined more precisely here.

Related concepts include best practices, voluntary commitments, frontier safety policies, norms, technical standards, etc. “Technical standards” has a more specific meaning, e.g. being codified by a “standards setting organization” – I think you could have “standards” in my sense without having technical standards in the typical sense (e.g., because it’s too hard to be precise about some things), and vice versa (e.g., technical standards that few people read/care about and which are paywalled, which is common today, unfortunately).

Regulation might make reference a particular set of standards or call for the creation of a set of standards. Some examples of potentially important standards efforts include the General-Purpose AI Code of Practice (called for by the AI Act, which I’ve discussed before here, here, and here) and the series of best practices being articulated by the Frontier Model Forum (FMF). It may be good for standards to evolve, with a tight feedback loop to daily practice, via some sort of private entities such as an industry body like the FMF, or via a regulatory market.

Incentives

My personal preferred path for incentivizing compliance with AI standards is for the US federal government to pass legislation that creates fines for violators of a specific set of safety and security standards (while also incentivizing honest reporting of mistakes), drawing on lessons learned from similar efforts at the state level and in the EU. This is preferable over a patchwork of US state policies as well as relying too much on the EU to set rules for American companies. I’d also like to see private actors drive compliance with these standards or variants on them (e.g., meeting a higher bar might be associated with lower insurance premiums). Multiple, independent “demand signals” of AI standards seems good to me from the perspective of avoiding capture and enabling a defense-in-depth approach, so we should not just focus on governments but also explore a range of market-based incentives, as well as incentives tied to specific high-risk industries (e.g., defense procurement).

Cyber insurance premiums over time (to be clear, it’s not obvious to me that cyber insurance is a great success story, though I think a range of mechanisms for incentivizing safeguards should be considered). Figure excerpted from this article by Swiss Re.

But not everyone agrees with me about that vision for the future, and that’s fine. Critically, you can support the basic idea of an AI governance triad while still preferring that we incentivize compliance in another way – such as via a governmental “carrot” rather than a “stick,” or through private carrots and sticks rather than governmental ones.

Examples of alternative demand signals for AI standards include:

• Government procurement requirements

• Company procurement requirements

• Investor due diligence on large capital investments and loans

• Insurer criteria that incentivize higher standards (for AI-specific insurance, cyber insurance, etc.)

• Consumer purchasing behavior, informed by private evaluations/scorecards (think “Consumer Reports for AI”)

• Requirements associated with importing restricted items (e.g., high-end computing hardware)

This is not exhaustive, and again, I don’t think we have to pick a single one. In my ideal world, there will be some convergence of standards (multiple demand signals for the same thing, in case, e.g., a company is more concerned with/affected by litigation risk than insurance premiums or vice versa), and some divergence (so we don’t have to choose between setting a bare minimum and pushing for continuous improvement).

Evidence

A company can say their system is safe and secure, or that they follow a given set of standards, but should you believe them? As AI gets more capable, the risks associated with mistaken claims, fraudulent claims, or too-vague-to-be-verified claims will increase, and a cloud of suspicion between competitors could drive corner-cutting (indeed, this is happening already and could get worse). Ideally, organizations building and/or deploying frontier will provide sufficient evidence to believe their claims.

Evidence that an organization is following AI standards can come in many forms. I think the question of how to ensure this evidence is compelling – including to very skeptical parties with very different goals, values, etc. – is a key one and merits much more analysis. But briefly, I would say that there are at least three key categories of evidence.

• A company can be transparent about the safety and security properties of particular systems or their whole organization, with this transparency either being voluntary or required. Some things can just be stated or shared, and easily verified by third parties (e.g., performance on a certain safety-related benchmark). But transparency won’t always be possible, and some information needs to be shared with a more limited set of parties, leading to the second category.

• External assessment of frontier AI can help in cases where outsiders don’t want to take a company at their word about something, but are OK with listening to an independent third party. External assessment can range from fairly shallow black-box evaluations which just use the existing product affordances (e.g., the API or a graphical user interface) all the way through to full-blown “AI audits” that look at normally-non-public information such as training data, internal documents, etc. Again, this is an area where much more could be said.

• A company can have a policy protecting (certain categories of) whistleblowers on safety and security issues, so that outsiders can have more confidence that if there were conflicting evidence, they’d hear about it eventually, or at least relevant government officials would. The stronger such policies are, the more “absence of evidence” (lack of visible whistleblowing or lawsuits) can become “evidence of absence” (regarding flagrant violations). There is a lot of bipartisan interest in protecting AI whistleblowers (at least those who specifically whistleblow to relevant federal agencies), and a growing set of external infrastructure to help AI company employees navigate related situations – precisely because the first and second bullets above are inherently imperfect. There has also been discussion of subsidizing whistleblowing in the context of export control violations. The basic logic behind whistleblower bounties, which have been successful in the financial sector, could also apply to frontier AI safety and security more generally.

Again, I prefer a defense-in-depth approach in this and many other areas. Ideally, the evidence provided by companies will be structured in some way that allows critical public scrutiny, such as in the form of a “safety case” for a particular AI system. System-level safety and security cases might be nested inside a larger “organizational safety and security case,” which makes reference to processes for external assessment as well as whistleblowing protections.

An example of a safety case that might be made for an AI system. Taken from here.

Related concepts include verification, third-party auditing and assessment, compliance reviews, structured access, safety evaluations, AI assessments, benchmarks, penetration testing, etc. Key frontiers include negotiating sufficient access for third parties to come to informed conclusions while protecting sensitive information, and demonstrating the safety and security of internal deployments. There are also various legislative efforts to ensure sufficient whistleblower protections for AI company staff.

Conclusion

This triad is not intended to be a complete description of current AI governance or a full prescription, either. But I think it has significant utility for concisely describing the situation we’re in today – e.g., as just one example, the EU’s AI Act articulates a particular set of incentives (e.g., fees up to some percentage of corporate income), these are associated with standards that are being hashed out in public via the General-Purpose AI Code of Practice, and there is debate about how evidence should work, e.g., what role third party assessment should play. The triad also provides a high-level roadmap for investments that entrepreneurs, executives, policymakers, and others can make – i.e., exploring new kinds of incentives, contributing to articulation of standards, and piloting new ways of providing reliable evidence.

Different people have different versions of this in mind, but I’d say the two key ingredients are:

• Access: The IAEA for AI would have access to (some subset of) the world’s compute supply chains and thereby know where the biggest datacenters are, much like the IAEA keeps tabs on enriched uranium. It would be able to inspect those datacenters to make sure GPUs aren’t going missing and that model weights aren’t being snuck out on thumb drives, and it would be able to audit the AI systems trained or deployed at the largest datacenters against some set of safety and security standards.

• Credibility: The IAEA for AI would have to be trusted by all (or almost all) parties as not only having expertise, but also being solely focused on ascertaining the facts. This would be accomplished through rigorous processes and professional norms and having a highly international staff. It would have to know what it’s talking about and also be seen as a neutral arbiter that is not “putting its thumbs on the scale” of a particular conclusion.

Despite the actual IAEA arguably having these properties to a large extent, and no one really doubting their top-line conclusions about Iran, bombs are still falling.

I’m personally a fan of the “IAEA for AI” concept and think that the nuclear/compute analogy is better than many people think (and actually, I think computing supply chains are much easier to monitor than nuclear ones – there is only one TSMC and one ASML, but there are nine and maybe soon ten nuclear powers).

At the same time, the current situation in Iran shows that even if an “IAEA for AI” is necessary for some purposes, it won’t be sufficient for resolving the tricky geopolitical issues raised by AI.

Aligned vs. misaligned interests

One reason that an IAEA for AI might be necessary eventually is that the world actually agrees on a lot of things related to AI, and could benefit from having a common set of baseline rules. People don’t generally want AI to go rogue, they don’t generally want for it to be easy to use AI to create biological weapons, etc.

Actually implementing seemingly-commonsense safety and security standards like this one may easier said than done, just as nuclear safety and non-proliferation are difficult in practice, but there is reason for hope based on the fact that some risks are ~universally objectionable. Screenshot from here.

So it may be possible to negotiate some set of bare minimum safety and security standards, and then audit everyone (at least everyone who has very large amounts of computing power) against those standards. This would give everyone the confidence that at least everyone’s doing the basics, and that no one has to completely throw caution to the wind in the “AI race.”

Breathing room in an otherwise very competitive situation is valuable, just as the IAEA helps to ensure safe nuclear energy production, and has avoided a Chernobyl-tier safety incident since Chernobyl, while also avoiding worst case security risks like diversion of nuclear weapons to terrorists.

But as Iran reminds us, just because most people have a lot of common ground most of the time doesn’t mean there won’t be conflict sometimes, and we should expect that to be true of AI, too. Iran seems to think (or at least be entertaining the idea) that they will be more secure if they build nuclear weapons even though it makes a lot of other people (especially Israel) feel less secure. An IAEA for AI, on its own, would not tell us what to do when someone doesn’t want to play by the rules or when the rules don’t address the situation at hand (e.g., because the rules have a “lowest common denominator” characteristic, only covering the very most egregious risks).

The last mile problem

As I alluded to above, people generally seem to agree on the facts of the situation in Iran – but not quite. There’s a “last mile” problem in the sense that there are some details of the situation, such as whether Iran has decided to actually weaponize their nuclear capabilities, which apparently the US and Israel disagree about. I’m not a nuclear expert but my rough sense is that some things are much easier to verify through inspections than others, and the “weaponization decision” part is trickier than the “large-scale uranium enrichment” part. I think it’s pretty likely that there will be analogies to this situation in AI.

For example, it may be very feasible to keep track of the top 10 largest datacenters and top 10 most capable AI models in the world since each one will involve multi-billion dollar investments. But it will be much harder to keep track of the top 100 datacenters or top 100 models, let alone the top 1000, etc. And it may also be very difficult to keep track of whether a company has disabled some critically important safety feature that they found was slowing down their effort to create an automated AI R+D loop.

There’s a concept in nuclear non-proliferation of “breakout times” – how long, once a decision has been made to build a nuclear weapon, until you actually have it, and how does this vary with the starting point, the technical capabilities of the country, etc.? The Iran nuclear agreement (JCPOA) was based around the idea of freezing Iran’s nuclear weapon potential in place and carefully tracking that they weren’t moving any closer to a weapon – achieving a breakout time of around a year. Longer is generally better since there is then more time to negotiate a resolution to an emerging crisis.

(For the avoidance of doubt, I want to note that I think it’s plausible we could have avoided the current crisis if the JCPOA were still in place. Even if true, the larger point is still true, namely that “the IAEA existing” is insufficient on its own)

Screenshot taken from here.

With AI, breakout times could conceivably be very short – it could be hours, minutes, or even seconds for some very specific but important change to occur. I don’t think this is self-evident, but it’s at least plausible. Perhaps the most dangerous activities will take months or more. We just don’t know yet.

Fortunately, the more capable and dangerous AI gets, the more capacity we’ll have to have highly intelligent and, critically, credible “AI auditors” helping us out. This raises the question: how feasible is it to automate an IAEA’s ability to keep track of that last mile, helping us “fight fire with fire”? And for what kinds of risks will this be feasible? I don’t think anyone knows, but I think people interested in AI governance should think more about this kind of issue.

The analogies that we use when talking about AI influence which aspects of the technology we pay particular attention to, and which solutions we develop and find tractable. Is AI like electricity, the printing press, the Internet, or nuclear weapons? Is it like none of these things, or like all of them in different ways? Will it be like one technology at one stage, and then like another technology (or even more like a species) at a later stage?

I don’t think there is a single best analogy, and there is no substitute for studying the actual technologies, impacts, and organizations involved in AI. But I want to describe one idea that I’ve been thinking a lot about lately, and which sheds some light on key debates in the field. The idea is that “AI is a liquid, not a solid.”

In literal terms, AI systems are trained by, and run on, solid machines (computing hardware). Those in turn are powered by electricity, cooled by air or liquid, and the ideas behind all of this are non-physical. But when I say that “AI is a liquid, not a solid” metaphorically, I’m trying to make a point about the amount of AI that exists and that people have access to, and about the ease with which AI spreads. Both of these properties have many implications in an AI policy context, so I think it’s useful for more people to internalize this analogy.

Amounts of AI

It’s natural to think of a system like ChatGPT (or Claude, or Gemini, etc.) as a single, well-defined entity that is sitting in the cloud somewhere, waiting to respond to your question. But this is an increasingly misleading way of thinking about AI. The amount of AI capability that an AI system has, or that an organization has, is a matter of degree, rather than coming in discrete chunks.

The amount of thinking that you get out of AI increasingly depends on how much you pay in order to get a (literally) more thoughtful response. At first this involved just paying more for a version of the system that would think for longer (in a “chain of thought”) before replying, but now this is becoming more like a knob you can tune – you can set an exact “thinking budget” depending on how much thinking you want.

Figure from this post on o1.

The “liquid-ness” of AI is also true at the level of an AI platform, like the OpenAI, Gemini, or Claude API. The number of copies of each model is in constant flux as companies adapt to demand. It’s a bit like a bathtub that has water going in from the faucet and out through the drain - the bathtub constantly fluctuates in terms of the amount of liquid inside it, and so does the amount of “AI” in a datacenter.

If you look at the field of AI as a whole, progress is also “liquid-like” in the sense of coming in small increments rather than sharp breaks. There was no single, sharp point at which AIs began clearly “thinking,” “reasoning,” etc., and before which they clearly weren’t. Similarly, whether AI systems are “human-level” depends on which humans and set of tasks you’re talking about, and also how much you’re willing to pay in order to run the system.

Measuring quantities of AI is more like measuring a liquid than counting the number of LEGO blocks in a pile, where there’s a clear right answer. As with actual liquids, you can try to come up with measurements and definitions in order to make sense of the situation, such as quantitative evaluations for AI performance, and some people define “AGI” in various more precise ways than others. You can also define what a quart or a milliliter is, and then say that you have a quart of water or a milliliter of milk, but at the same time, scientists still recognize that these are ultimately human-made categories, rather than being inherent units of water or milk.1

Another liquid-like aspect of “amounts of AI” is that you can split up or combine different “AIs” or components thereof in many ways, just as you can transfer the contents of two containers of liquid into a bigger container, or vice versa. There’s a wide variety of ways that this can work. For example, AIs can be “chained together,” where the output of one becomes an input to another. AIs can also be “distilled” or “quantized” into a smaller version that is cheaper to run but a bit less capable, by training the smaller model on the outputs of the bigger one. You can also sometimes literally stitch together different (parts of) neural networks into a larger composite network.

There are a few policy implications to this liquid-ness of AI quantity:

• This metaphor underscores why we should not wait for arbitrary milestones to be achieved in order for us to take AI seriously – the liquid is all there will ever be. AI is not going to freeze or crystallize into some solid, well-defined “AGI” tomorrow. We have a lot of AI already, and we are getting better at making more of it, drop by drop every day. That will be the case for the foreseeable future. There’s always a better model or system waiting to be built, there’s always a smaller one to be built and run more cheaply, and there’s always a way to run more copies of an existing one.

• This framing also underscores the fact that we should not expect any company or country to have a monopoly on AI capabilities. Some liquid containers are bigger than others, but that is different from someone having a monopoly. There is no specific threshold — like the formation of a crystal — that only one organization can achieve. Because leadership in AI will always be a matter of degree, we should assume that if one company is capable of creating one kind of system, many others will be able to do so soon after, or even at the same time by spending more computing power to run worse algorithms. This has many implications I’ve discussed elsewhere, including the need for a degree of cooperation on safety and security between different AI efforts.

Access to AI

In addition to AI quantities being liquid-like, AI access is also liquid-like.

Liquids have a tendency to leak out of containers that aren’t well-constructed. Insights about AI technologies and products also often leak out if companies aren’t careful, and they are also easily shared by people who want to do so for licit or illicit purposes. More generally, AI developments tend to diffuse outward from their point of origin by default, absent rigorous efforts to contain them.

Sometimes we should actually apply the rigorous effort required to contain AI capabilities, such as when protecting frontier AI model weights from theft and tampering. Other times we should just accept and embrace the fact that AI gets everywhere and lean into the many benefits of that. But either way, our policies and our overall mental model of the technology needs to reflect AI’s ease of spreading.

Different levels of sophistication in stealing or tampering with AI systems. Currently, I’m not aware of any company that even claims to be able to reliably protect themselves against attackers above level 3, and most are not capable of defending at that level. Taken from this summary of this report.

When an AI model is released open source,2 it is not only copied many times but also modified in various ways, like a puddle spreading in many directions simultaneously. Likewise, it only takes a very small “crack” in a corporate “container” for an idea to spread far and wide, even if it isn’t accompanied by full details or model weights. OpenAI’s announcement of o1 was immediately followed by a huge array of new research projects in many countries, including DeepSeek’s R1-preview and then later R1, despite OpenAI having provided only a very small amount of information in the original post. Just showing that something is possible, and roughly what technical direction is required, is often more than enough to motivate talented teams to follow in a leader’s footsteps, and sometimes match or exceed them.3

Graph from DeepSeek’s R1 paper, showing a similar approach to what OpenAI demonstrated a few months prior — reinforcement learning to elicit progressively longer and more useful “chains of thought.” See also this post.

Some very simple ideas in AI are worth millions to billions of dollars, in terms of the computing power they could save or the capabilities they could help reach. But different aspects of AI have different “viscosity” and propensity to spread. Hardware spreads less easily than algorithmic ideas – China has spent many hundreds of billions of dollars to catch up in semiconductor manufacturing, and has made progress but is still behind. One way of thinking about this is that building hardware requires getting many millions to billions of ideas just right, whereas once you have the hardware, the actual process of training a particular AI system involves dozens to thousands of ideas.

The higher “viscosity” of hardware, compared to ideas about AI algorithms, plus the liquid-like nature of AI capabilities, means that having a single copy of a model doesn’t mean that you can run it at the same scale as the organization you got it from. Stealing o1’s model weights would help you make a rogue o1 API, but it would not be sufficient.

One of the facilities being built as part of OpenAI’s Stargate initiative. A lot of hardware goes towards training, but a lot of hardware is also needed to run AI systems at scale. Even the biggest companies struggle to meet demand. Photo taken from here.

There are a few policy implications of this part of the analogy:

• We should plan for the rapid diffusion of AI. We currently rely heavily on the safeguards applied by a small number of frontier AI companies in order to keep society safe (e.g., enabling easier creation of biological weapons). That’s not a sustainable strategy, because those capabilities will spread to other actors, and the more actors with a given level of capability, the more varied their goals and values will be, and some will be very malicious and reckless. The current situation is like having a bunch of delicate electronics and paperwork right next to a big water tank that’s about to burst. We should start much more aggressively AI-proofing many parts of society.

• In parallel with planning for some inevitable leaks and spreading of puddles, we also should at least make sure we know how to contain AI systems when we really need to. We should do a much better job of protecting some AI systems from theft, tampering, and misuse – see here and here (and the links therein) for further discussion. Developing frontier AI in the companies that are currently doing so is like storing acid in a bathtub – the acid will eat right through the bathtub, and needs special containers, instead.

Conclusion

There is no perfect analogy for AI.

In order to understand AI, the most important thing is to use it. See what works and what doesn’t, get a sense for how this varies across different products, how the inputs and settings change the outputs, how the speed and performance of cloud-based AI differs from what you can fit on your computer, how performance is changing over time, etc.

But humans think analogically, and I hope this post helps a bit in improving the clarity of your thinking. I also hope it encourages you to be a bit more skeptical when you hear someone use phrases that suggest AI is something more solid than it really is (e.g., “racing to AGI” or “an AGI monopoly”).

Acknowledgments: Thanks to Larissa Schiavo for helpful feedback.

Frontier AI systems — state-of-the-art neural networks and the infrastructure built around them such as input/output filters and access controls — require especially strong governance. They are currently being abused in harmful ways such as assisting with fraud and cyberattacks; they will soon make it much easier to carry out novel biological weapon attacks; and sufficiently capable AI systems could resist human oversight and either seize, or be used to seize, enormous amounts of power. Yesterday’s frontier systems quickly become today’s mundane tools, but at any given time, we should pay particular attention to frontier AI systems.

Urgent action is needed, and we shouldn’t let perfect be the enemy of good. But we need to make sure we’re heading in the right general direction. This post is about one way in which we might be off course.

Most (though not all) thinking on AI safety, security, and policy is framed in terms of mitigating the risks of individual AI systems. But we also need to be thinking much more about organization-level governance, which counterintuitively will often be the best way of addressing issues with those individual systems. This is a major difference in framing: less like “Is o3 or Claude 3.7 Sonnet safe?” and more like “Will OpenAI or Anthropic cause catastrophic risks?”. For several reasons I describe below, we need to start asking the latter type of question a lot more, and it is not reducible to the former.

Recapping the model → system transition

Today’s focus on “AI systems” is already an expansion of an earlier framing – “AI models.” The shift from focusing on models (the actual neural networks themselves) to focusing on the larger system that includes those models was an important one. This recognizes the fact that there are many layers to a product like ChatGPT, Claude, or Gemini. There are filters on inputs and outputs, the models use tools (e.g., browsing and code execution), and there are various access controls (e.g., fraud prevention for new accounts and rate limits).

The shift from models to systems was particularly important in order to pave the way for the latest paradigm, in which it matters a lot how much computing power is used to run the model (i.e., how much “test-time compute” is applied), such as by making the model think for longer or running multiple copies in parallel.

Taken from here. How a model is used as part of a larger system, including how much test-time compute is used, matters a lot.

Particularly for “closed models” – in which the model weights are not released openly but instead are accessed via an API or a product – it matters a lot, safety and security-wise, what the system’s features are.

A smaller, narrowly focused model might scan for malicious inputs to a bigger model, or there may be limits on how much you can do with a free “burner” account in order to make it harder to scalably abuse the technology (e.g., to generate large amounts of spam or to carry out cyberattacks). It also matters which “system prompt” is being fed to the model on a given day, telling it how to behave. These are all system-level questions, not model-level questions.

Models matter, and the point about the importance of systems can be overstated. But overall this was a good transition to make, and below I’ll just use the term “system” in a way that includes “pure” models as a special case. But we still need to zoom out further.

Systems are often too small and narrow of a focus

Models and systems are critical to apply safeguards to, but an exclusive focus on them will blind us to many aspects of AI governance. I am not the first to make this general point, which has sometimes been called a “[corporate] entity-based” approach to AI governance, though I haven’t seen a sustained argument for it. This section will only scratch the surface but I hope it gives a rough sense of some reasons why the system-level focus needs to be augmented with an organization-level perspective.

Security is important in order to judge the overall impact of a system, and security is largely an organizational property.

Today, the strongest argument for AI systems being safe is often an “inability argument” – essentially, that the system is too stupid to be able to do anything very dangerous. That kind of argument won’t work much longer, because AI is continuing to progress very quickly, as all the leading companies are saying as clearly as they can without coming off as too alarmist.

Taken from Epoch’s dashboard here.

As we move into a world where inability arguments don’t work anymore, the safety measures we place on systems will become more critical, and so will efforts to prevent those systems from being stolen or tampered with. It’s often reasonable to proactively release model weights despite this meaning that the safeguards can be undone, but there’s no reason to think that this will always be the case – sometimes society will just not yet be pandemic-proof and it will be irresponsible to “democratize” biological weapon creation right away. Indeed, many in the industry are concerned we’re heading towards such a world in the near future, and In order to even have the option of holding back certain dangerous capabilities (versus them being stolen or leaked), you need good security.

Figure taken from a blog post here, describing a longer analysis by RAND here.

Unfortunately, security isn’t moving as quickly as I and many others would like, but for our purposes here, the key point I want to convey is that security is often best thought of at an organization level, not at a system level. Thus, as external stakeholders evaluate the level of risk that an organization might be imposing on the rest of the world, you can’t focus exclusively on the properties of particular AI systems.

For example, consider whether a company invests in patching its software quickly (even when this is annoying to users), or whether they allow employees to access their work accounts on personal phones – these are organizational questions. These organization level steps will significantly affect how easy it is to (e.g.) spearphish an employee, or otherwise gain access in order to steal model weights or algorithmic insights.

The impact of using an AI system often depends on context, which can only be understood at the organization level.

AI systems are inherently “dual-use” – they can be used for military or civilian purposes, and for offense or defense. That’s just the nature of intelligence, and fortunately, most people aren’t evil, so AI gets used in many beneficial ways. But there are some bad people, and AIs might themselves be bad – and the nature of bad activity is that it’s sometimes subtle, with many small actions that look fine on their own adding up to a larger malicious plan.

In order to understand whether a particular set of “tokens” (outputs from an AI system) are harmful or not, context matters. And the relevant context will extend beyond the system (and outside the literal “context window” of the model). For example, suppose that there is a large amount of traffic going through an AI API about discovering vulnerabilities in an open source code library. It matters whether that traffic is coming from the maintainers of that library, with the aim of proactively finding bugs and fixing them, or from employees of a government agency that aims to exploit any vulnerabilities they find in cyberattacks.

Examples of activity detected on OpenAI’s platform. Taken from here.

Telling one from the other is not always trivial. And taking the time to understand such distinctions is an organizational question. Suppose, for example, that the company has an incident response team that notices suspicious traffic from a network of fraudsters and sometimes gets in touch with governments for input on whether to take down the account immediately or to let the activity continue in order to gain valuable information about how the network operates. Or suppose the team does research on the email addresses used to create the accounts being used – research that is done outside of the AI system itself. Whether such an effort exists at all, what societal goals it serves, and how effective it is at achieving those goals are all organizational questions (perhaps you could say it’s a “platform” question, but it’s clearly different from a “system” in the narrow sense that is typically used).

Organization-level governance is more tractable due to its relatively slower rate of change compared to model and systems.

The features of AI systems change constantly, and there is always a new system on the horizon. In contrast, while organization policies and culture also change over time, they change more slowly, making them a more tractable “target” for governance.

Consider the safety and security policies that frontier companies have committed themselves to, such as Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and DeepMind’s Frontier Safety Framework. I believe that these have been updated publicly at most four times (in the case of Anthropic), whereas during that same time period, each company has launched more than 4 systems, and each of those has often been changed many times.

Version history for Anthropic’s Responsible Scaling Policy, taken from here.

Assuming that third parties (such as third-party testing organizations and regulators) have limited capacity, it makes sense to focus that capacity on a smaller number of targets and to do a better job at assessing each target. A recent paper that I contributed to describes one piece of this puzzle: what it might look like for third parties to review evidence (e.g., from internal documents and staff interviews) about whether a company is following their stated policies. This is in some sense what the general public ultimately cares about (whether a company is staying within some set of specified risk tolerances), moreso than the properties of any particular system.

That doesn’t mean ignoring systems entirely – I still think there’s a role for system assessment as part of a larger plan for assessing organizations. I don’t think you can truly understand whether a policy is being followed without digging into the details at least some of the time. But relatively speaking, I’d like to see more of a focus on whether companies are following their stated policies, which, if positively answered, cuts across all of their systems. I expect there to be a lot of efficiency gains from taking this perspective versus the “whack-a-mole” approach of waiting for a new system launch to be imminent and then conducting a rapid analysis.

An organizational perspective is more likely to “catch” certain categories of risk, such as those from internal deployment.

Taking the system-based whack-a-mole approach, where external involvement is only “triggered” by a forthcoming launch, may lead to limited or no external involvement on entire classes of risks, such as those associated with risky internal deployment.

Frontier AI systems can still pose dangers even if the organizations building them don’t deploy them in commercial products. A company might, e.g., not give access to their most sophisticated model to users outside the company, but instead just use it to produce new scientific insights that are then leveraged to make the next generation of systems. Or they may “distill” the smarter system into smaller, more efficient systems that are deployed externally, but use the smarter system directly themselves. Whether this is a responsible thing to do may depend on the company’s internal safety and security practices, whether external experts and government officials are being appropriately apprised of and consulted on the process, whether critical safety lessons learned are being shared with other frontier companies, and so forth. And these questions are more likely to be asked and satisfactorily answered if the relevant “lens” that we take as a society is a company’s activities overall, rather than just the properties of the very latest system they have put on the market.

Taken from Apollo’s report here.

System-level governance will fail in the hardest cases due to “the Volkswagen problem.”

The hardest cases for AI governance might be those where the stakes are high (e.g., where there is a lot of temptation to cut corners because safety measures impose significant cost) and where there is mistrust between the actors involved (e.g., between two militaries would ideally prefer to be cautious in deploying AI in combat situations to avoid catastrophic accidents but don’t trust the other party to be cautious, or a company that believes a regulator is out to get them and can’t be trusted with full information).

Taken from here.

In these situations, we may run into what I call “the Volkswagen problem,” a reference to the Volkswagen emissions scandal, in which emissions results were falsified. Essentially, the system was designed to behave differently when it knew it was being tested.

This is a perennial concern in many regulatory contexts as well as many arms control contexts: if someone has strong reasons to be deceptive, you can’t just look at a technical system in isolation. You have to look at the wider context (e.g., Were trucks seen leaving this nuclear facility in the hours leading up to an inspector’s arrival? Were engineers pressured to cut costs and stick to an unrealistic schedule?).

For many governance purposes, it’s reasonable to take companies at their word that they’re showing the public, regulators, companies, etc. “the real system.” But that won’t always be reasonable, and AI is likely to be very high stakes. So in at least some contexts, we need to move into more of a “trust but verify” mentality. This will be especially important in the context of verifying any future international agreements on AI. Addressing the Volkswagen problem can by definition only be done at an organization level rather than a system level, since the very question at stake is whether you’re looking at the right system.

“System” is rapidly becoming too nebulous of a concept, anyway.

While it’s not easy to verify an organization’s adherence to its own policies, or more generally whether it is creating significant risks, at least an organization is a reasonably well-defined entity. A company is a legal entity with an org chart, a headquarters, and a mailing address. There’s a lot of diversity among organizations, but they are in some respects fairly standard and coherent categories, with clear “hooks” where different governance tools fit in (such as safety and security processes, organizational cultures, whistleblowing policies, etc.).

AI systems, on the other hand, are undergoing a Cambrian explosion to such an extent that it’s not clear whether anyone knows what they mean by it anymore, and what counts as the same system versus a different system is fairly arbitrary.

Consider a specific model, OpenAI’s o3. When o3 was first announced, there was not a single set of test results for it, but rather different results for different levels of test-time compute. The low-compute system has different performance than the high-compute system, though they both involve the same underlying model. Later, o3 was first deployed at a system-level, as well, in the form of the Deep Research agent.

In some sense, Deep Research is a good target of governance – it’s a named product and there is a system card describing (an early version of) it.

OpenAI’s description of the different safety mitigations applied to Deep Research.

But there’s no reason Deep Research has to stay the same over time, and using that term as if it’s a coherent entity can be misleading. The system referred to as Deep Research on Day 1 was just considered good and safe enough to be deployed, so it was deployed, but it could be improved. Some months in the future, perhaps OpenAI will bring a new datacenter online and consider offering Deep Research Max, which will use much more test-time compute. The question of whether Deep Research Max would be the “same system” as Deep Research is, ultimately, an arbitrary one, whereas “is OpenAI imposing significant risks on the world through its work?” is what we ultimately care about.

Of course, the organization level of analysis is not an excuse to ignore complexity. Systems are still included within the scope of the organization, and can’t be ignored. But organizations are a key part of looking at the situation in the right way, and the organizational lens will often be the key to shedding light on risks we would otherwise miss or misunderstand.

Conclusion

There are many ways of slicing and dicing AI governance. There may be important levels of analysis between systems and organizations (such as platforms), and also levels of analysis above organizations (such as countries, and of course ultimately the whole world). But organizations are my current bet for where we should be ramping up our focus the most right now, while we continue to iterate towards mature internal and third-party assessments of AI systems.

I’ll say more in the future about how all of this fits together – how systems “ladder up” to organizations, and how organizations “ladder up” to the world. As a sneak preview, I think we need to begin thinking about “organizational safety cases” that build on and subsume system level safety cases while adding additional elements such as accounting for how compute is being used, sharing rigorous independent evaluations of how robust security controls are, and providing evidence (via, e.g., whistleblowing policies and evidence of a strong safety culture) that any significant policy violations will likely be found and fixed. And we then need to ask how much of the world’s compute is “spoken for” by organizations with strong organizational safety cases, and what this means for society’s exposure to residual risks from “ungovernable compute.”

As we move towards an integrated, multi-level vision for AI governance, we should also learn from successes and failures in somewhat analogous cases like regulation of “systemically important financial institutions" with many financial products and electrical utilities with many power plants.

Acknowledgments: Thanks to Larissa Schiavo for helpful feedback on earlier versions of this post.

You can find it here.

The article argues that — even while the US competes vigorously in some areas — American policymakers also need to recognize that China will inevitably have powerful AI capabilities. Given the many risks of AI, the US thus has a strong interest in cooperating to ensure that each countries’ AI capabilities are developed and deployed responsibly.

Thanks to all at AI Frontiers for the smooth editing and publishing process, and others who gave feedback on earlier drafts.

Private governance of AI, as Dean envisions it, has a lot of appealing properties. And given the urgency of AI policy, it’s worth giving a serious hearing to plans that might have “legs” in the current political environment. So I was excited when Dean published more details on his vision, and I considered writing a long response to it.

But since I have several other time-sensitive writing projects right now, I decided to take an 80/20 approach instead: I made some miscellaneous comments on a Google Doc version of Dean’s post, he responded to some of them, and we’re sharing the incomplete dialogue below.

Now here’s the crazy part: the link below should work for everyone, and in order for everyone to see the comments, it also has to be the case that anyone can comment. This could be a disaster, but maybe it will lead to an interesting debate. Click on the link and comment at your own risk (e.g., if you’re logged into a Google account, consider whether you’re OK with everyone seeing that, and if not, log out and be anonymous instead).

The current dialogue in the doc does not arrive at any conclusion. I’m not even sure that I endorse everything I said, since Dean had some very thoughtful replies, and I was mostly in quick reaction mode rather than “having a coherent opinion on the proposal” mode. And conversely, there are a lot of things I would love to say now in response to Dean’s responses, but I need to move onto other stuff right now, so we’re leaving things unresolved.

Hopefully this post serves its intended purpose — inviting you to think open-mindedly and critically about Dean’s proposal and private governance more generally, and fostering more public discussion about it.

I would suggest reading Dean’s post in full (and subscribing to his Substack!) before checking out the doc, since even my early comments below assume you understand the basic idea.

Normally I say something like “curious to hear what people think” but then don’t actually let people comment on my blog, but this time, I will really hear what people think! Please be nice to me, Dean, and others in the doc, or I’ll have to shut this experiment down.

https://docs.google.com/document/d/1ftDGn83gCqoc45Z_vQ0mcuPNukdb3DYBx5fsQw6PLZI/

This is a reasonable perspective, since securing anything against sophisticated attackers is in fact super hard. Attackers have a bunch of advantages (like only needing to succeed once and having many possible ways in), enormous resources (the Stuxnet attack reportedly cost around a billion dollars to carry out), and very strong incentives to succeed.

Ultimately, I am not a doomer myself, but in order to understand why, we need to first understand where such folks are coming from.

Note: while sometimes people (myself included) use the term “security” to refer to various things, including preventing misuse of AI, I’m here just going to be talking about information security.

Why Doomerism Is Appealing

Information security is hard

There are countless examples of information security failures – because it’s a hard problem.

Chinese hackers stole millions of extremely sensitive security clearance applications from the US’s Office of Personnel Management (OPM). US critical infrastructure is so thoroughly penetrated by attackers that it will reportedly take years to even fully understand how bad it is, let alone fix it. Despite Iran taking very serious steps to keep the Natanz uranium enrichment complex secure, Israel and the United States still sabotaged it. For a longer list, see Appendix A in this report.

The apex predators of the security world. It’s very hard to defend against these. Image from the report linked above.

Information security in general is hard, and protecting AI systems in particular is also hard. AI software and hardware are super complex. There are many dependencies in the AI supply chain, making it hard to even understand, let alone safeguard. Each company uses products, data, hardware, and code from many sources. Everything is constantly changing and breaking. Employees will get annoyed when you ask them to do fairly standard things like use a company phone or use 2-factor authentication for everything, let alone something more severe like not bringing electronics to the office. And you can’t always trust your employees, who might be spies or coerced by third parties to compromise your company.

Security is costly in AI

As alluded to above, people at companies generally don’t love implementing new security measures. Why? Because it’s expensive – not so much in terms of direct costs (though you do need to hire people, buy some hardware/software, etc.), but rather, it’s expensive in terms of slowing things down. Of course, if you invest too little in security and have a major incident that destroys the reputation of the company, that will really slow you down later. But that’s a concern for later, and people aren’t great at planning for the long-term. It’s also not totally clear that there will really be costs for the company later due to bad security. Maybe the company gets hacked but the public never finds out, so users and national security could be damaged while the company pays no direct financial or reputational price.

What does this look like concretely in the case of AI companies trying to improve their security while also competing on the market? If security is dramatically ramped up in a unilateral fashion by a company, it will slow down research and slow down shipping of products. I’ll return later to ways that the costs can be decreased through collective investments in e.g. R&D, but the point here is that there are in fact costs. For example, let’s say you want to protect model weights from theft by making it so that fewer employees can access them. That means that, all else equal, it’s harder for some people to contribute to the project. And if you start compartmentalizing information about AI models and algorithms, fewer “eyeballs” are on various research problems, potentially slowing progress.

There are also AI-specific vulnerabilities, like distillation attacks, in which an attacker takes a shortcut on creating a similar model by “distilling” the intelligence reflected in your model, by training on outputs from the model rather than the weights themselves. It seems hard to defend against these without making your product less useful. And there are many different things to protect – not just model weights (which we’ll discuss in more depth later) but also product plans, customer data and customer identities, algorithmic insights, code, etc. Oh, and people move between companies all the time, bringing secrets with them.

Costs matter

It’s not just that there are costs, but that companies are, rightly, cost-conscious.

No company is more than a few months ahead of their closest competitor in the current “main arena” of competition (large pre-trained language models fine-tuned with reinforcement learning, and then run with a lot of inference compute). Some have bigger leads in very specific areas but not in the main arena. These companies are in my view a bit further ahead of DeepSeek (let alone other Chinese companies) than some commentators appreciate, in large part because of compute. But now that DeepSeek is (informally) China’s national champion on AI, that could change quickly as a larger fraction of Chinese compute gets steered towards them.

Graph from DeepSeek’s R1 paper, showing a similar approach to what OpenAI demonstrated a few months prior — reinforcement learning to elicit progressively longer and more useful “chains of thought.” See also this post.

Given this competitive situation, if a security intervention slows a company’s research or product development down by, say, 50%, or would cause researchers to be so annoyed that they all leave to join a competitor down the street (or worse, leave the country), it’s simply untenable from the company’s perspective. Most security interventions don’t have that big of an effect, but if you did everything at once it would be quite disruptive, so companies are gradually ratcheting things up bit by bit, doing the most that they think they can afford – which is not going to be enough to prevent sophisticated nation-state attackers from stealing the extremely powerful systems that will exist very soon. Indeed, this is acknowledged by the companies in question, which, depending on the company, either set their ambitions lower in the near-term (i.e., they acknowledge they are not there yet but are working gradually towards it), and others don’t even aspire to defend against sophisticated attackers.

Additionally, some of the “classic” approaches to achieving extreme security are simply not viable in the context of AI (e.g., requiring everyone involved in frontier American AI companies to have a security clearance is not viable given how many non-citizens are critical to the functioning of these companies). And AI research requires conversation, access, dynamism, etc. in a way that may be different from (e.g.) keeping a well-defined, decades-old nuclear weapon system humming. In that sense, the “costs” of applying some extreme measures in an AI context are potentially infinite because they could eliminate the very advantage that is purportedly being protected.

Given all of this, it is reasonable to ask – is it really worth it to protect AI systems against the most sophisticated attackers, or will that just slow things down to the point of being counterproductive? Maybe the best we can hope for is something like mutually assured destruction, as some recently argued.

The case against doomerism

Security is hard, it’s costly, and the companies on the frontier can’t afford the huge costs that would be required to secure their systems against all possible attackers — at least if they’re trying to sort it out themselves, and if they are aiming for perfection (e.g., never leaking algorithmic ideas, product plans, etc.). But I think we can set our sights on something more tractable, and work together towards a solution that protects (part of) the most important systems.

The question is not whether we should apply very rigorous standards of security to all systems but whether we should do it sometimes – and to figure out how to do it at the lowest possible cost. I think we should, and that the most realistic (and still useful) goal of such efforts is protecting model weights.

Model weights are weighty

When GPT-4.5 was released recently, OpenAI staff joked a lot about how it was a “chonky” model. That is, it has a lot of parameters, so running it requires a bunch of GPUs to make a bunch of calculations. Model weights are not the entirety of an “AI system,” which may comprise an interface, some tools, various ways of really pushing those model weights harder and longer to get better answers than the model’s first intuitive guess (i.e., ways of “applying test-time compute”), etc. But, the weights are at the center, and I think they will sometimes be worth protecting, and relatively soon.

GPT-4.5 is probably not the chonkiest model that there will ever be, and based on public speculation about the size of GPT-4, it probably has many trillions of parameters. But let’s be conservative and pretend that it is just a trillion parameters, and that models won't get any bigger. If that were true, then models would be a few terabytes in size, depending on the precision.

Screenshot of this tweet.

If you were to write down each model parameter one by one on a piece of paper, and could write one per second, it’d take hundreds of thousands of years. Of course, computers and cables can transmit such files much more quickly than humans can write them down. But not infinitely fast, and they can also be designed with “speed limits” on certain types of activities. Given that almost all uses of AI – even by researchers at these companies – do not require direct access to the model weights, this is a very promising fact. It means that we can potentially design systems that are useful (allowing many forms of access to AI systems) while being secure in at least one specific sense (making it very hard to exfiltrate the weights from a datacenter). This is not by any means trivial but I have yet to see an argument as to why it’s impossible.

One response to this is that model weights aren’t that important anyway–didn’t DeepSeek replicate o1 with just a few million dollars? Well, the total cost (not just the training cost) was definitely way higher than that, but let’s pretend that it was. GPT-4.5 certainly cost much more, and will certainly be much more capable when reinforcement learning is applied to it at scale, compared to DeepSeek’s current models. Eventually, DeepSeek will be training models that cost hundreds of millions of dollars or billions of dollars just in terms of marginal training costs – and by then, US companies will (assuming the US government doesn’t back down on export controls) be even further along.

The important thing is that there will always be some model weights we want to protect very well, whether they are ones that cost billions of dollars, or ones trained on very sensitive data or performing very sensitive tasks, etc. Whether some weights aren’t worth protecting is separate from whether we should build the technologies to protect some weights.

AI training is getting expensive, and training runs will soon cost more than the cost of executing the Stuxnet attack (if they don’t already). Taken from here.

Similar points can be made about model distillation as a possible counter-argument to investing in weight security: what is the point of protecting the weights if you can just distill the model by sampling from it? Well, first of all, I’d note that the impact of distillation is sometimes overstated – even if one granted that DeepSeek’s R1 was partly based on distilling from o1 (which I haven’t seen conclusively demonstrated), it was still not as performant as OpenAI’s best models on various benchmarks. The science of distillation is still fairly early but it seems to me that there will probably be some cases in which distillation will not be enough and an attacker will still want to get the actual weights (for example, perhaps this happens in cases where it matters a lot what the base model is). In those cases, we should know how to defend model weights.

Security does succeed sometimes

It’s been a while since San Francisco or New York got nuked. Actually, they never have, despite there being a large number of really bad folks out there who would love to. Why? Because nuclear weapons are pretty well protected and controls on their use are pretty good. It’s not literally impossible to steal them or for someone to launch a nuclear missile illicitly, and there have been some close calls. There was a pretty close call at the end of the Cold War, when the US and Russia worked closely to lock up nuclear weapons while the Soviet Union collapsed. But generally speaking, major investments in security have paid dividends, and made it so that it’s more cost effective to just build your own (or, at least, building your own is less likely to get you killed than trying to steal one, though still somewhat likely to get you killed).

You might say that this is not analogous to AI, because nuclear weapons aren’t being used all the time, so of course it’s easy to keep them secure. Well, yes, that’s true, but a primary concern of nuclear planners is making sure that if and when they are needed, those weapons will be available. That’s how nuclear deterrence works, and again, lots of people spend a lot of time making sure this is the case (while also safeguarding these weapons from theft and misuse). Like, literally billions of dollars and hundreds of thousands of person-years have been spent on this problem. So it is in fact possible to reconcile usability with security.

But can security be done for something used by billions of people? Yes. Look at cryptography, for example. Private keys that, if stolen and misused, would be worth billions of dollars (e.g., those of root CAs), are routinely protected from theft. As a result, public key cryptography undergirds a global system that generally allows people to send emails and buy products and services securely online.

From Cloudfare’s description of the steps they take to secure their most sensitive stuff. From this blog post. See also this from ZCash. Sometimes security is worth it.

I don’t mean to downplay the difficulty of security. Sometimes really damaging attacks happen, and smaller ones happen constantly. A huge amount of critical infrastructure in the US is cooked and this would be exploited in the event of a war.

But typically the way these attacks happen is that someone exploits an issue with the software, or “hacks” the people involved, rather than, e.g., infiltrating a deep vault with a super secure safe at the bottom with a USB stick on it. My claim is that with the right software and hardware design, we can make model weights more analogous to the deep vault scenario, while still letting “good” traffic go in and out of the vault through well-understood channels.

Weights are chonky, and also, like private keys in cryptography, not everyone actually needs to directly access them (and indeed it can be counterproductive for everyone to have direct access to them). APIs for OpenAI, Anthropic, and DeepMind provide users with the ability to sample from and even sometimes fine-tune AI models, without taking them. Sometimes these APIs are designed in the wrong way and actually you can steal (parts of) them, but that seems like a solvable technical problem. Between good software and hardware design, it seems possible to make AI systems that are useful for a very wide range of purposes while making the costs of stealing them much greater than the costs of recreating them.

For further related discussion, see Appendix B of this report.

Costs can be driven down and shared

As discussed above, costs matter in AI. But that doesn’t mean everyone should have to fend for themselves.

In fact, that’s part of why governments invest in research and development – companies may not fully capture the benefits of such investments so they may not invest as much in them as is ideal for society as a whole. Breakthroughs in science and technology can benefit everyone, and should often be subsidized. Indeed, in the nuclear context, this happens all the time – countries that have more sophisticated security controls for nuclear weapons will help other countries to secure their own stockpiles, because (pretty much) no one benefits from loose nukes.

Also, different security tools and practices can get easier and cheaper over time through innovation. Widely used security tools like single sign-on (SSO) make things both more secure and easier for users, and while there are probably some irreducible costs to very good security in AI, we won’t know how cheap we can make it until we try.

You can just do things (and thereby make them cheaper for others to do them). From this page. We don’t really know what the experience curve for weight security looks like, but the best way to find out is to do more research and pilots and share what’s discovered.

So, again, I am on the side of optimism regarding the (partial) solvability of AI security. With government leadership and different companies contributing to different parts of the problem and publishing their results, the overall “waterline” of the sector can be driven upwards, at much lower cost than if everyone had to figure it out on their own.

Two areas in particular jump out as priorities for urgent investment:

• Confidential computing features: This class of technologies is considered by many experts to be a key building block of weight protection, but by all accounts the features available today are not scalable to production-size model weights, nor has there been sufficient testing of them to have confidence in robustness against serious attackers. Several steps could potentially help here: government and philanthropic support for related research and development, tax credits or other incentives for chip companies to speed up the timeline of making this technology viable in production settings, and a commitment to consider the availability (or lack of availability) of such features in future government contracts. There should be widespread and well-resourced testing of these systems before going into production to the extent possible, or it will be difficult to have confidence in the lack of vulnerabilities.

• Open source, well-documented secure API designs: Companies (and national security agencies using AI) will always have a need to build some custom security software internally, due to the quirks of their particular software and hardware set-ups, we shouldn’t overestimate the similarities between AI systems across companies, either. Governments and philanthropists could thus support the development of well-maintained and well-documented software libraries that demonstrate how to provide internal employees with the ability to sample from and fine-tune models while still having strong security guarantees against exfiltration, and this investment could improve security at many companies simultaneously. It is not uncommon for a company to use the exact same model architecture (e.g., because they have fine-tuned an open source model), and there are many high-level similarities between models at different companies (e.g., using autoregressive Transformers and using some kind of mixture-of-experts approach). Additionally, companies often use some of the same software, like Kubernetes, which predictably introduces various security risks, and public resources to defend against such risks could be very valuable especially to less-resourced AI developers and deployers. Comprehensive coverage of all possible security situations that a company might encounter is impossible but you could imagine, e.g., a resource that provides worked examples of securing Llama weights on GPUs, a generic mixture-of-experts model on TPUs or Trainium chips, etc., and AI developers and deployers could take inspiration from these examples rather than starting from scratch.

A realistic vision of success

I suspect it will be too costly to try to protect a big fraction of algorithmic secrets. They are easy to “save” (in one’s head) and transmit (by saying a sentence or two to someone). But those algorithmic secrets are only valuable because they are used on computing hardware (which is relatively governable) and because they allow you to make better model weights or use those weights more effectively. And most computing hardware is used for running model weights, not training them. At the end of the day, it does matter who has the most capable weights and who doesn’t. And I think we can achieve weight security for the most capable models.

What would this (partial) security success story look like in practice? For details on some of the technical challenges involved, see the RAND report discussed above, but here’s a very rough personal sketch:

• Frontier AI companies have very high confidence that weights won’t leave their datacenters without multiple parties signing off on such a transfer. Depending on the sensitivity of a particular model, this “high confidence” can be arbitrarily increased by toggling the “security vs. convenience” knob. This level of security is verified repeatedly by highly-resourced penetration tests.

• Cables going in and out of the datacenter are heavily scrutinized for tampering, and there is software and hardware on both ends of each cable guaranteeing that only certain kinds of data and instructions can flow through them. For example, perhaps there are certain “white-listed” queries and responses that can go through at a certain volume each day, and anything not fitting this data structure will trigger a careful review.

• Extensive physical security would be applied at the datacenter (“guns, guards, and gates”) – while chips would be designed in such a way that no one should be able to steal the weights through direct access, defense-in-depth will still be needed to be really sure. For example, no one should get anywhere near the chips without several people knowing exactly what they’re doing and watching their every move.

Cables at a datacenter - taken from here.

Obviously you don’t want to do this for all AI systems (which I’ll revisit in the conclusion shortly). But sometimes you should do it, and we need to figure out how to do all of this as cheaply and efficiently as possible for the few cases where it’s needed.

Also, I’m not saying that in the ideal case, no one should ever protect anything other than weights. Of course a company should protect user data from at least non-state attackers. “Normal security” is critical for frontier AI companies as well as companies applying AI in critical applications. And some algorithmic secrets are probably worth protecting for at least some period of time. The question is just what, if any, large-scale, cross-industry and cross-sector moonshots we should be betting on in the context of AI information security, and I think weight security is a good target for that.

Conclusion

I want to again underscore that security is hard and costly and that costs matter. Not every AI system should be under lock and key. We get lots of benefits from some models being open sourced, for example, and I don’t personally want to have to jump through a zillion hoops to use ChatGPT.

But we need to think of secure AI as an investment that will be needed for some purposes, and which is currently not moving at the speed of AI progress. I’d like to see cross-sectoral work to drive down the costs of very good weight security as quickly as possible.

If you work at an AI company, try to be part of the solution to security. I’m not saying you shouldn’t complain when something your company does is actually “security theater” that isn’t worth the effort – but conversely, there may be things your company could do that are worth it, and maybe you should speak up for your security team from time to time and see if there are engineering problems they’re working on that you can help with. And then see if you can share the results with the world, driving down costs for others.

There’s a lot of active work in this area and I’ll share more about it — and ideas for policies that could speed things up — in future posts.

Acknowledgments: Thanks to Lennart Heim, Sella Nevo, Carly Tryens, Jason Clinton, Katherine Lee, and Larissa Schiavo for helpful comments. The views expressed here are my own.

Below are some things I’ve decided not to focus on in my own projects this year (though I may advise others on them here and there). They’re all super important, but I think my skills/knowledge etc. are better suited to working on other problems. This is far from an exhaustive list of important AI-related topics to work on, of course, but maybe one of them is the right fit for you?

Helping people “feel the AGI” more

As discussed here (and in my writing generally), I don’t think we have much time left to sort out AI safety, security, and policy. The biggest bottleneck on getting these issues solved might turn out to be the extent to which people viscerally appreciate the gravity and urgency of the situation. Judging from conversations I regularly have with people in civil society, journalism, and the deep state, I think we’re still pretty far from most people appreciating the situation we’re in. Even some people at frontier companies seem to think they’re just building cool products and not something more analogous to (though not exactly the same as) electricity, or even a new, smarter species.

There are potentially many ways to push on this. Non-exhaustively, this could include clear writing that takes the pace of progress seriously; creating and giving demos of increasingly powerful and dangerous capabilities; coming up with better ways of visualizing, interpreting, and extrapolating AI progress; producing science fiction portrayals of AI that reflect significant expert input; producing documentaries; and probably various other things.

Technical infrastructure, social norms, and legal clarity for agents

I think some companies overstate the importance of AI agents, since you don’t need AIs to be particularly agent-y in order to have huge impacts. Even without a big push on agents, things will get crazy this year “just” due to much smarter chatbots, so calling it the year of agents feels off to me. But at the same time, agents are indeed coming and we don’t have the technical systems, societal norms, or legal clarity required for governing them.

What are the AI agent equivalents of stop lights, railroad tracks, etc. – infrastructure that we need in order to keep a powerful new technology “on the rails” while reaping its benefits?

Chan et al. use the term agent infrastructure to refer to “technical systems and shared protocols external to agents that are designed to mediate and influence their interactions with and impacts on their environments.” (from this recent paper). We need to sort that out quickly. One area that I’d particularly flag as essential is personhood credentials, which will be important both for distinguishing between humans and agents without violating privacy, as well as delegating to agents when appropriate. But there are many other things that need to be built.

We also need to be talking more about what exactly the role of “humans in the loop” should be in different economic and social contexts, and who is legally liable for major incidents. There are ideas here and there and there, and there was very briefly a big discussion about some of the liability stuff during the debate over SB 1047, but then it died down… we never solved the issue, though. Many US states are trying to propose their own path forward, which is creating a mess (which could have a silver lining of accelerating federal action, as long as that action doesn’t block reasonable risk mitigations).

More people should be making sure this all lands well, both in the short-term and the long-term.

Economic impacts of AI – near-term disruption, solutions, and endgames

I’m quite confident that AI will eliminate many millions of people’s jobs in the next few years. Some people disagree, or agree but think that those people will find other jobs quickly. This is not the place to sort that disagreement. In my view, this is a very important topic both for its own sake (unemployment causes severe harms) and because it will affect the larger political situation around AI.

Among other priorities in this area, there should be more forecasting of near-term “disruption hot spots” (for example, I’m very concerned about call center workers in developing countries, though there are likely many others, including in the US). Additionally, there should be more discussion of and action on policy solutions, as well as the long-term “endgame” for paid employment, education, and leisure. If AI indeed enables explosive economic growth in the near-term as I and many others expect it will, everyone could “retire early” with a high standard of living. Do we want that? How do we avoid a WALL-E scenario, or worse, while also giving people more control over their time? What does a path towards a good outcome look like?

The art of the EU AI Act deal

I’ve so far written two posts on the EU AI Act, one with Dean Ball and one on my own. I’m still planning to engage on this topic to some extent, and in particular I may write another post when the third draft comes out soon, but I won’t be giving it my full attention. More people should, though.

Regardless of whether you take more of a glass half-full or glass half-empty perspective on the Act generally or the Code of Practice specifically, it’s pretty high stakes. Simplifying a bit, there are three possible outcomes: the Act gets killed or gutted by a coalition of US tech companies and the Trump administration putting pressure on the EU; it gets implemented as-is with a Code of Practice that emerges from the normal negotiation process, and — either out of genuine concerns about compliance costs or to make the US administration happy — one or more AI companies pull out of the EU; or some secret third thing. More people should be thinking about the secret third thing.

This might look like a grand bargain between the US and EU that allows the US to claim a win (avoiding the worst excesses of the Act from going into force), while avoiding the EU having wasted several years. For example, there could be an amendment that repeals one or more portions that are considered especially objectionable to US companies (e.g., on copyright) or there could be some sort of explicit interoperability introduced between the EU side and the US side (e.g., allowing compliance with some US-backed private standard to serve as a substitute for Code of Practice compliance).

These are just rough ideas for what a deal could look like. But the point is that, at least unless and until the US is on top of things, we should be very concerned about the most serious effort at AI governance either being totally abandoned without a replacement, as well as such efforts falling short of their potential to actually raise the bar on safety while being efficient to implement. Contrary to some people’s beliefs, companies do not in fact have sufficient incentives to mitigate all major risks, and competition is driving corner-cutting that needs to be reined in somehow.

AI literacy

As I have discussed before, I don’t think AI companies have done a good job of explaining how their technology works. Misconceptions abound, and both systematic over-use and systematic under-use are common as a result. That’s pretty concerning from a “wanting people to make good decisions” perspective and also a liability for companies and policymakers as the technology starts to cause serious accidents, and then people look around and realize the companies on the forefront didn’t really try particularly hard to avoid such outcomes.

There have been some piecemeal efforts by non-profits, governments, and companies, but the growing literature on the topic shows that even very educated people routinely fall for hallucinated AI outputs, so something isn’t getting through. Onboarding instructions remain non-existent or minimal, with no companies (to my knowledge) having provided any empirical evidence that their user interface design improves outcomes in terms of appropriate reliance. And there is starting to be evidence that overuse may atrophy users’ critical thinking skills. Not ideal!

This is notably different from feeling the AGI, though not unrelated. One can look at the trendlines and conclude “hmm, seems exciting/concerning” without having much of an explicit or intuitive understanding of how to actually use AI effectively day to day, and vice versa. Ideally we’d be strong on both fronts, though.

Some of the main themes were roughly in line with what was expected (e.g., discouraging overregulation, emphasizing the upsides of AI, underscoring the US’s commitment to leading in AI, and raising concerns about Big Tech and authoritarian governments imposing a limited set of values on people through AI). I’m not going to provide a full summary, and would recommend that you just watch it here (his remarks start at about an hour and 15 minutes in).

Below, I will highlight a few themes and nuances that were harder to predict since they are areas where the DC consensus isn’t yet clear. In some cases I was encouraged to see these bits, and in other cases I have concerns about what these comments may portend.

AI security

One of the reasons I was in Paris to begin with was to attend a side-event called the AI Security Forum. So it was heartening to hear Vice President talk about the need to protect both AI technologies and semiconductors from theft and misuse. By distinguishing between AI technologies and semiconductors, he presumably meant to refer to protecting model weights and/or algorithmic secrets and code. As discussed elsewhere, I think security is in some respects “the new safety” in that it’s something that has been neglected for far too long, requires urgent investment due to long lead-times (e.g., for secure chips and datacenters), and should in theory have bipartisan support due to its non-ideological nature. While safety has gotten a bit of a bad reputation in some circles, which I’ll return to below, I think security may have more staying power.

I hope that Vance’s remarks are a sign that there will indeed be a bipartisan consensus on this issue, since I’ve heard a lot of concern about it from folks on the left, as well — no one wants American AI to be trivially stolen by China and Russia (other than China and Russia). There are big challenges in actually executing on the vision he outlined, and the rubber will meet the road in, e.g., how security requirements are baked into federal contracts, execution on (or replacement of) the late Biden executive order on datacenter permitting (which tied energy and security together), and more generally, the US government sending a very strong signal to industry that this is an urgent priority.

Safety and regulatory capture

While this requires reading between the lines a bit, one plausible interpretation of Vance’s remarks is that he thinks safety regulations (if any) should be very lightweight in their implementation, targeted on the most extreme risks, and should not be captured by Big Tech.

The emphasis on only extreme risks can be inferred via process of elimination (i.e., the other risks he explicitly or implicitly minimized), and the concern with efficient regulation can be inferred from his discussion of European regulatory overreach. He didn’t quite explicitly say that he’d be open to new regulation of any sort (or, acceptance of existing regulations such as the datacenter conditions discussed above, which to my knowledge haven’t yet been repealed), but he did say that safety can’t be entirely thrown to the side, which was in some respects a stronger statement (actually acknowledging there’s a “there there”) than the leaked summit statement.

As discussed elsewhere, I think that some AI safety regulations aren’t quite as onerous as some believe — e.g., I think the AI Act’s Code of Practice is trending in a fairly reasonable direction as it gets revised, though I also share some of Vance’s concerns about overly bureaucratic processes and broad scope. And I also agree that we should be wary of industry capture — indeed, the ability to be taken more seriously in what I say on policy is part of why I left OpenAI. But I think it’s very important to bear in mind that it is not just Big Tech saying that there are serious risks coming in the very near-term that we need to be prepared for. And while vigilance about regulatory capture is sensible, we also need to take seriously the fact that CEOs and company staff don’t have super strong incentives under the current administration to raise the alarm about catastrophic risks and yet they continue to talk about them, and there is now a very credible international report on them which seems to have been well-protected from industry capture.

Workers having a seat at the table

As I and others have discussed elsewhere, there are some interesting politics on the horizon at the intersection of AI and jobs. Vance was notably optimistic in his remarks about the impact of AI on the economy, and emphasized that American policy will aim for AI being a tool for rather than replacement of humans. But this is easier said than done and while I’m running out of time before my flight, I’ll just say very briefly that having helped build a team focused on this topic, there is no plan to ensure such an outcome right now. It is notable that Vance sides on the economic populism side rather than the tech right end of the spectrum on this issue, and said explicitly that we should not just think of AI as a disruptive technology but that it will and should be well-integrated with the world of atoms, rather than just occurring in the world of bits. Again, what this means in practice is TBD.
__

Sorry I couldn’t say more or edit this more (have to run for my flight), but I hope some folks find this helpful.

The Chinese company DeepSeek’s recent release of R1 – which is now the most capable open source AI model – generated a lot of concern among American technologists, policymakers, and investors. Robust responses, especially tightening of American export controls on GPUs, are necessary given the increasing importance of AI to the global economy.

But I worry that amidst all the discussion of what’s new about R1, there has been too little discussion of the ways in which it’s just the latest chapter in the modern history of AI. That story started several years ago in 2018, around the time I was working at OpenAI on open sourcing the (now primitive) GPT-2 language model.

GPT-2, R1, and nearly every other AI model released since 2018 have all been part of a consistent story: AI capabilities that rival and ultimately exceed human intelligence are easier and cheaper to build than almost anyone can intuitively grasp, and this gets easier and cheaper every month. This lesson has deep policy significance. It means that even while competing vigorously with China for relative influence, the United States must come to terms with the fact that it will not have a monopoly on superhuman AI capabilities.

Scale and skill

If recent history in AI has taught us anything, it’s that “all” you need to develop AI systems that meet and ultimately exceed human capabilities are two ingredients: millions to billions of dollars worth of computing resources and data (scale), and dozens to hundreds of highly talented scientists and engineers (skill) applying that scale to the training of artificial neural networks.

In the first scaling paradigm, researchers and engineers train a big neural network to predict what comes next in a sentence, by trying to complete billions of sentences billions of times, and tweaking the parameters of the network after each mistake in order to get better. As a result of this process, AI models learn grammar, knowledge about the world, and rudimentary reasoning skills.

During my time at OpenAI, I saw that time after time, “just” scaling this paradigm up – an enormously skillful endeavor – would consistently yield improvements in performance, and, counterintuitively, improvements in the performance per dollar (e.g., if you train the same size model for longer, you are getting more value in the long run when you run it to serve users).

Each and every time, it took at most a few years but typically months before other companies would achieve the same level of performance using a broadly similar, and always relatively simple, recipe. Moreover, late-comers would typically replicate the achievement for much lower costs than what OpenAI originally spent, because they could build on all the tips and tricks that the global scientific community published. Ultimately these savings would flow back to the companies with the most computing power when they did a later training run, yielding 90%+ cost reductions on the OpenAI API over time.

Starting last year, OpenAI, DeepSeek, and other companies have started to show that scale also can lead to enormous improvements in the ability of AI systems to think through problems step-by-step in a long “chain of thought.” By trying over and over to solve complex problems in areas like math and coding, and getting rewarded for each success, o1 and R1 gradually learn reasoning techniques like breaking problems down into chunks and checking one’s work.

Graphs from DeepSeek and OpenAI, respectively, each show different aspects of scaling up the new paradigm. The graph from DeepSeek shows performance improving over the course of training, and improvements when more attempts are made to solve each problem. The graphs from OpenAI show improved capability as a result of applying more computing power during the training phase and during the inference phase, respectively.

The benefits of scale are why well-enforced and up-to-date export controls on GPUs are more important than ever. In fact, to a greater extent than with the last paradigm, reasoning models like o1 and R1 are hungry for computing power at the time of use (not just at the time of training), which is probably why DeepSeek ran into issues serving their app to this new wave of millions of users.

But the fact that skill and scale are all you need also means that there is no secret ingredient that the US can protect forever in order to have a monopoly on superhuman AI. AI capabilities will always be a matter of degree, and what you can accomplish with a given amount of scale is constantly changing.

Given what DeepSeek’s skilled researchers and engineers could achieve with (at most) tens of thousands of high-end GPUs, it’s clear that they will accomplish incredible things in the coming years with the greater number of GPUs they will get, a growing fraction of which will be domestically produced over time. Their ability to achieve these capabilities with fewer GPUs than their American counterparts required skill, but also is a continuation of the previous trend of ever greater performance and efficiency.

If the US maintains leadership in computing power and skillful research and engineering, it may have the best models and be able to serve them at a larger scale than China can. It might even be able to create dramatically better models than China, if automated engineers applied to AI lead to an “intelligence explosion.” But even second-place status in AI will confer very powerful military and economic capabilities to China, given just how easy AI has turned out to be. China is too far along in their development of semiconductor manufacturing, and they are too committed to attaining advanced AI capabilities, for them to be stopped entirely, short of a war that would be ruinous for both sides.

Competing while coexisting

With other commentators, I agree that Congress and the White House should ensure that the Bureau of Industry and Security has the funding, technical support (e.g., from DOGE), and political support that it needs to implement export controls effectively and update them rapidly as needed.

AI competition needs to be tempered with a recognition that China isn’t going away as a player in AI, and that coexistence must also be a part of America’s strategy. Even if China doesn’t surpass the US in scale, it has more than enough skill and scale to do a lot, and will be strategic in how it deploys the resources it has. This may look like, for example, putting particular effort into releasing efficient open source models or into military AI systems. Ironically, China may actually be better positioned than the US to dominate open source since Chinese companies will, because of export controls, naturally be thinking more in terms of efficiency than their “GPU-rich” American competitors.

Since China isn’t going away on AI, the United States will have to learn to cooperate with them in areas of common interest such as measuring and mitigating dangerous capabilities (e.g., models being misused to create biological weapons), sharing lessons learned in the area of AI alignment (to avoid very sophisticated AI systems evading human control), and negotiating (and ultimately verifiably enforcing) basic rules of the road on military AI to avoid unintended escalation of conflict. Again, this happened with the last wave of AI scaling, though with smaller stakes than the next wave: as competitors caught up with and in some cases surpassed OpenAI, there was a clear need to share lessons learned and develop common norms around safety and security.

Some have said that R1 is a Sputnik moment, and I agree, but this is true in a very specific sense: Sputnik did not demonstrate overall technological superiority of the Soviet Union, but rather the feasibility of reaching orbit given a serious effort by a fairly technologically advanced country. The US, China, India, and other countries also gradually took advantage of this new domain, and while competing, they still had to coexist in space (e.g., avoiding hitting each others’ satellites with debris). Rather than just being a domain of military competition, satellites served a wide range of national, international, military, and civilian purposes. The same will be true of AI.

There are signs that President Trump recognizes this: in his comments on DeepSeek, he said it was a wake-up call for Americans, but he also applauded DeepSeek on their achievement and said that it was in everyone’s interest to have cheaper AI. He’s right – in the same way that it’s in everyone’s interest to have cheaper satellites, as long as they aren’t crashing into each other.

Acknowledgments: Thanks to Jordan Schneider for helpful comments and for inviting me to the ChinaTalk podcast where I made some related points on DeepSeek.

In a previous blog post that I co-authored with Dean Ball, we provided feedback on the first draft of the General-Purpose AI Code of Practice (COP). See that post for more context, but to briefly recap, the COP is a way of complying with the EU AI Act. The second draft of the COP was published a few weeks ago.

Some of what we said in the first post still applies to the second draft, and some doesn’t, but I won’t repeat any of that content here. Instead, below I will first summarize my overall view of where things are at in the process, and then share fresh reactions to the second draft.

Note that the bulk of this post will be pretty inside-baseball-y and will assume some knowledge of the COP, which can be found here, though maybe you should wait for the third draft if you’re just starting out. I won’t be offended if you don’t read all of this, unless you’re one of these people. I suspect most of you will probably be interested in the big picture section, though.

Also note that I am not a lawyer and didn’t have time to consult one on this so it’s possible I may be misinterpreting some things, and of course this is not legal advice regarding compliance.

The Big Picture

The chairs and vice-chairs iterating on the COP will hear many different perspectives on the big picture. Some will say that the COP is a case of tyrannical overreach from European bureaucrats that is impossible to comply with, and others will say that it’s a toothless document that caves to industry and VC lobbying.

Here’s my take, which is somewhere in between.

Generally, things seem to be moving in a direction of greater specificity and coherence, and reducing both overreach and under-reach, between the drafts. This suggests that the feedback process is working reasonably well and there is a genuine desire to balance various factors. I also would add that many aspects of the current draft are very well-considered, though I won’t enumerate all the things I like about it below (e.g., I think it’s the right call to allow external assessors who aren’t EU citizens).

At the same time, it has clearly been hard for the chairs and vice-chairs to keep up with an insane schedule, and I really wish the European Parliament had just asked to kick this process off much sooner, like when the Act was first passed, instead of a few months out from it going into force. This rushedness shows up in, e.g., there being a lot of unfinished business in each version that seems like it’s just a matter of time rather than requiring more feedback. I don’t think there’s a solution to this now, but it’s just worth being aware of so frustration is directed in the right place (i.e., at Parliament, not the chairs and co-chairs).

With one exception, big companies should not have too much trouble complying with the provisions on “models with systemic risks,” the part I’m paying closest attention to. Models with systemic risks are, for now, basically models that require more compute to train and are therefore assumed to pose higher risks of various kinds, and I’ll circle back below on the future of this framework below.

Since only a small number of companies can afford to make models with systemic risks as currently defined, it is reasonable to have at least slightly higher compliance burdens for these kinds of models, though it shouldn’t be so extreme that even big companies can’t comply. It looks like we’ve avoided that failure mode, at least – my assessment is that it should take no more than one additional program manager, lawyer, and engineer to comply with these provisions if a company is already taking reasonable actions to assess and mitigate major risks (which most relevant companies have already committed to doing anyway via voluntary commitments). Perhaps it will be less than this, and I will consider ways to make things more efficient below. And perhaps it’s more, but it’s certainly not several dozen, again assuming a reasonable starting point.

However, just because compliance is feasible for big companies doesn’t mean that the provisions will achieve their original goals. If compliance is inefficient or mis-targeted with respect to the real risks at stake, then the AI Act could be counterproductive by giving safety a bad name within companies and distracting employees and governments from attending to more pressing issues. I’m not saying that this will happen, since I think the COP will continue to improve, but I’m just contextualizing why I think it’s worth helping to make the COP as efficient and effective as it can be.

A recurring theme below is that whether the COP makes sense depends in large part on the threshold for determining that a model has systemic risks. I assume below, unless stated otherwise, that the provisions on systemic risks will apply to only a small number of companies, but I revisit this issue in a few specific cases.

Lastly, I’ll note that the working groups are clearly struggling to manage some fundamental tensions baked into the AI Act, particularly on open source. I don’t think that these tensions are resolvable by the AI Office, and either require different solutions from other political bodies, or an amendment to the AI Act down the road.

The biggest tension, in short, is between having requirements be proportional to risk, on the one hand, and treating small businesses and open source as basically untouchable, on the other. In general, yes, bigger companies will have bigger and more risky models, and yes, more compute-intensive systems/companies deserve a closer look, and yes, it’s good to avoid crushing open source. But baking these loose correlations into the AI Act as immutable axioms leads to all sorts of strange contortions, like saying that someone open sourcing a model with systemic risk to the EU as a whole should not require anyone on staff red teaming it, but that those who deploy that same model via an API (where there are more affordances for preventing harm, including stopping deployment or changing the model) should have to do some red teaming before deployment. Isn’t that exactly backwards?

This is what happens when there are multiple conflicting constraints built-into the legislation as a result of lobbying from various directions, and this should be a cautionary tale for the US going forward.

Diving into the Details

Change over time

Both the first and second drafts have noted that there is not yet a plan for updating the COP over time.

I hope that the third version begins to shed more light on this, and would like to suggest one possible answer:

• The COP should be updated quarterly, though there should be a grace period such that it’s OK to be in compliance with a version that is up to one quarter old. Less frequent updates than this would lead to being way out of date relative to the state-of-the-art, and more frequent updates (or a lack of a grace period) would make it hard to plan ahead.

• The update process should consider not just changes to the COP but also the trigger for systemic risk, since these cannot be cleanly separated. If needed, Parliament should pass legislation allowing these to be jointly optimized (right now, the chairs and vice-chairs are designing the COP in isolation which is tricky).

• As is currently happening with the drafting process, the AI Office should provide indications of “where their head is at,” including possible future changes being contemplated. This would appropriately thread the needle between avoiding big sudden changes, on the one hand, and avoiding any big changes (even when they’re needed), on the other.

• The feedback process should include, in addition to Futurium, a secure email address to which any party (including those who haven’t registered as stakeholders) can send input. Additionally, there should be at least quarterly opportunities to register as a stakeholder in Futurium or any other higher-bandwidth modes of engagement.

• There should be some sort of baked-in presumption in favor of simplification and lightening the compliance burden, to counteract what will likely be a default tendency for the rules to grow more and more detailed over time. This could include, e.g., a target page length of less than 50 pages, or relying as much as possible on best practices codified via another body, a presumption in favor of deleting or having to justify certain rules on a yearly basis, or various other approaches.

Inefficiency in reporting processes

As noted in the big picture section above, there are various ways in which compliance could be feasible but still not ideal, such as by being inefficient in a way that gives negative associations to safety and policy (and the EU), or focusing on the wrong things. I have some concerns of this nature when it comes to the documentation required for explaining development/deployment processes at a company generally and for specific models.

In short, I think the EU should make explicit that compliance can occur via public documents that substantively address all or most of the required information, and for companies to simply supplement this privately and/or publicly with specific outputs for EU purposes, as opposed to creating entirely separate documents from scratch including all of the information required. Already, companies do a lot to produce “system cards” (as well as API documentation) that address a large fraction of the topics covered in the COP, often due to voluntary commitments. It would be inefficient to require all of this information to be rewritten in very slightly different ways, and right now it’s not clear whether this is permissible.

Alternative compliance options

As discussed elsewhere, I think it’s valuable to explore ways of verifying compliance with AI rules that leverage the judgment of a human “on the ground” moreso than formal documentation and technical mechanisms (though all of these have a place). One way of doing this in an EU context is to have external assessors be given deep access to a company (e.g., by being “employees for a week” every few months) and to report to the EU that they found the company to be in compliance and either documenting the details themselves or providing them just upon request. Essentially, the idea is to substitute “the human touch” (and someone who will over time have a lot of experience assessing compliance and writing it up) instead of company employees doing paperwork. This option may be particularly important for small businesses which may struggle to parse the long COP document and prefer a more lightweight option, though I would also like to see innovation in compliance processes at big companies.

Recall the concern about giving safety a bad name within companies, and generally making things worse. While having a third party poking around at the company may not be everyone’s cup of tea, it may be the better option compared to an employee (who should perhaps be working on the substance of safety rather than its documentation) doing the same, and a third would get more efficient at this over time compared to someone who does it part-time and intermittently.

Note that this is sort of a substitute for the recommendation above on “inefficiency in reporting processes” – if there isn’t a burdensome reporting process to begin with, then this idea is less important, but there may still be value in exploring this kind of approach even if the efficient reporting problem is solved. For example, consider the case of a small company that does not have experience writing system cards and that might want help getting started, and then repurpose the documentation produced by a third party assessor.

As discussed further below, a single assessor may not be able to cover everything at least in the near-term, so it may not be literally one employee for a week, but perhaps a few people with expertise in different areas could join a company for a week and thereby make things more efficient overall, since again, assessors will be quite familiar with the COP if they do this routinely.

Over and under-inclusive reporting requirements

As noted in the first post, the details of the reporting requirements in the first table seem not fully justified, and I won’t repeat all of what was said there, but will revisit one theme.

Some things that should be public are not very explicitly called out in the second draft, such as a system’s “goals” (e.g., as reflected in its “constitution,” rule-based reward models, etc.). And conversely, some things that are called out, like synthetic data details, are not obviously appropriate to demand in full as they can be highly sensitive. Likewise for forecasts of future capabilities. It has not yet been demonstrated that the European Union can protect such IP from theft, and unless/until that is demonstrated, there should be a recalibration of what information is requested in areas that aren’t specifically called out in the AI Act and which are not clearly required in order to assess safety risks.

If and when the EU demonstrates a strong capacity to protect information (e.g., having a SCIF set aside for this purpose and clear standards for screening of personnel who would receive the shared information), and there’s a stronger case made that full information on methods is essential for the AI Office to do its job, there should be more of an expectation of sharing information in such secure settings. But for now, I believe the EU should (unless statutorily required) generally be requesting transparency on capabilities of AI systems, goals of systems, evaluations of risks, and mitigations for those risks, not methods for producing the capabilities to begin with, or forecasts of future capabilities. Sometimes these are closely intertwined, but sometimes they are not.

Note that this is one of the recommendations on which I am most unsure. I could see a good argument for (e.g.) it being in the public interest for current and expected future capabilities to be fully open, though currently this seems risky to me, and the COP would not achieve this currently – it would just require it to be given to EU authorities, not the public. It seems to me for now that, at least unless and until company security has been dramatically improved and compute governance essentially solved (such that there’s not much anyone can do to “weaponize” information about enticing capabilities, such as by stealing them or scaling up to replicate them), then there should be some discretion in what to say about current and future non-commercial capabilities.

As is, it seems to me that sharing forecasts with the EU in an insecure way will just create a insecure repository of information at the EU AI Office that bad actors will use to decide whom to hack and for what exactly. I also think policymakers are not currently bottlenecked on information about the extremely fast pace of AI progress, on which the evidence is abundant about the overall big picture trajectory. But I could be wrong about some of these things.

Summing up, in some areas there is a clear public interest in knowing more than the COP currently requires, e.g., what AI systems are “optimizing for.” And conversely, some of this information should probably not have to be disclosed as it adds little to the EU’s ability to make good decisions, while potentially exposing sensitive IP.

Responsibility for misuse

In the bullet on the risk of “large-scale, illegal discrimination,” the COP says that “this risk category does not include the intentional misuse of models.” If I understand correctly, the intent is to avoid platforms from being held responsible for bad actors doing bad things with their platforms. But if we’re talking about illegal discrimination happening at a large scale, then (besides being illegal) that would be a violation of any platform’s terms of use. And elsewhere in the COP, it says that after deployment, companies should have “safeguards to prevent misuse.” So I am not sure what this caveat is accomplishing and it seems misguided to me.

What about fine-tuning?

The COP says that “During training and post-training enhancements (e.g. fine-tuning, reinforcement learning, or similar methods, but only when done by the model provider themselves or on their behalf), Signatories commit to assessing and potentially mitigating systemic risks at pre-defined milestones at which the model could potentially become significantly more capable.” I don’t know what the “but only when done by the model provider themselves or on their behalf” part means. Who is responsible for mitigating risks of an API-based fine-tuning service? It seems to me that a big chunk of the responsibility, if not all of it, should be on the part of the company making that service available, since they know the most about the model’s risks and are in the best position to implement safeguards.

Incentives for innovation on deep access

I was excited to see the following: “To facilitate model elicitation, Signatories commit to working towards giving more model evaluators, internal and external, grey- and white-box access to their general-purpose AI models with systemic risk, considering a cost-benefit analysis with regards to model security (see Measure 12.2).”

“Working towards” such deep access is hugely important, and relates to some things I’ve discussed elsewhere. But how quickly are companies supposed to work towards this? With no KPIs and no clear incentive to do more than the bare minimum here, as well as the complexity and multi-dimensionality of the problem/opportunity space being explored, it’s not clear to what extent this provision will get any traction.

I’d like to suggest that the AI Office think about this as something to incentivize, such that if you allow deeper access in one area, you are “rewarded” by a lighter paperwork burden or some other benefit. This relates to (and can be combined with) the point I mentioned above about using external assessors as “employees for a week.” Perhaps the COP could say something like “In recognition of the benefits to the larger ecosystem of innovation in this area, proactive innovation in this area and public sharing of lessons learned will be recognized by the AI Office as substituting for documentation of [Measures XYZ].” I’d have to be more intimately familiar with the COP to have good suggestions for Measures XYZ.

Tool and best practice sharing

Measure 10.8 says that signatories will make “state-of-the-art evidence collection best practices widely accessible to relevant actors in the AI ecosystem,” and gives KPIs with specific numbers of actors to share this information with. The intent of the measure is to help the wider ecosystem, including small and early stage businesses which will presumably not have the social connections to know how to get in touch with big companies in order to benefit from small-scale information sharing. So it would be more efficient and better for the ecosystem for such information to simply be published openly, and for big companies to somehow avail themselves of smaller and earlier stage companies for assistance that is not well covered in such public materials (e.g., doing office hours periodically or having a publicly-shared email address for reaching out with questions).

Security details

It’s great to see more granularity in the discussion of security this time around, though I have a few uncertainties about the current details.

In particular, the COP states that signatories should implement and advance research that could help prepare for SL4 security (discussed here; see also this post), if they expect to be targeted state-level adversaries. But what should this expectation be based on? I’m not sure anyone has a great answer other than “prepare for the worst,” which isn’t very actionable for smaller companies in particular. It’s possible that in some cases, SL3 will be overkill, and likewise, in some cases SL4 may not be enough.

As with many other details in the COP, much depends on the threshold for models with systemic risk. I’d suggest the following solution for jointly adjusting the COP, the security details, and the systemic risk cutoff: expectations of whether to expect certain categories of attackers should be based on the estimated budget of those attackers in relation to the cost of the IP in question. If, say, the systemic risk threshold corresponds to a billion dollar budget, then companies should be planning to ultimately get to SL5. If it’s much lower, then a proportionately lower level of protection makes sense. This is a very rough sketch, but may be a good starting point for a solution to this issue. Additionally, there should be some sort of explicit plan (whether reflected in the COP or not) for the EU to harmonize its decision-making with US and Five Eyes threat intelligence.

Pooling resources

There are several areas where the COP says that companies should do things that I don’t think they should be doing on their own for efficiency reasons. For example, research on SL4 (mentioned above) should ideally be done in collaboration with other companies, academia, governments, etc. Other examples include, non-exhaustively engagement with external experts and stakeholders, which requires skill and investment to do effectively; anonymous reporting channels, which could benefit from common infrastructure (e.g., like this); bug bounties, which also may benefit from shared infrastructure; and support for external scientific research.

It would be valuable to explicitly call out the permissibility and encouragement of collaboration in these specific cases, and/or in a cross-cutting manner in one of the introductory sections.

Safety versus non-safety resourcing

I mentioned above that there was one exception to my claim that it is feasible to comply.

The main exception my claim re: compliance requiring up to 3 marginal employees (for companies with mature safety/security efforts already) is the set of KPIs for Measure 10.4. These prescribe that the “[a]verage percentage of engineering hours spent on model elicitation for model evaluations used as evidence for the highest risk tiers as compared to the largest internal non-safety project [should be] 75%.” And then there’s a lower number for lower risk tiers.

That’s a super high percentage – basically this means that companies either need to nearly double their engineering staff, or dramatically decrease their effort on “non-safety projects.” I can see where they got this number – I suspect there was a mistaken assumption that the largest non-safety project can’t be that big, since companies have a lot of different projects, so how big can 75% of it be? But this misses the point that for big companies, projects are not equally sized. The largest non-safety project may involve dozens or even hundreds of people working on (e.g.) the next frontier model’s pretraining and post-training.

Also, I think the whole idea of a clean distinction between safety and non-safety projects is incorrect and harmful, since these sorts of things are and should be blended together in practice.

So I would just suggest scrapping these KPIs entirely and reconsidering what makes sense here.

Proportionality violation

As alluded to in the section on the big picture above, the COP authors are struggling with inherent tensions baked into the AI Act by legislators. Nevertheless, it’s worth stating explicitly that in many cases, the KPIs for SMEs and open source models are much more lenient for no clear reason grounded in actual safety and security considerations.

As just one example, consider the KPI for “Percentage of internal staff hours spent on exploratory safety research as part of all research hours,” which is 10% for non-SMEs and 0% for SMEs and/or providers who exclusively release open-source models. Even those who think that open source models are in some sense inherently safer, or who are worried about crushing small businesses, would, I think, be hard pressed to coherently justify an infinitely lower red teaming burden.

Incentive alignment on KPIs related to assessment results

It’s great for companies to actually be in compliance, and for assessments to accurately conclude this. But it’s bad if they’re judged to be in compliance and they’re actually not. If assessment results can’t be trusted, the whole system falls apart, so it’s important to consider the incentives created by KPIs very carefully.

This is why I worry about the lack of qualifiers on KPIs 15.2, which states “Adequacy and adherence assessments yield a positive result.”

It seems likely to me that many assessments will, and should, yield mixed results, and companies should not be encouraged to bias the process in a way that yield an unambiguously positive assessment (to be clear, such biases do not require any malicious intent on the part of companies, and may occur in small and subtle ways that add up over time to a watered down system of compliance). I would suggest, then, that there at least be some sort of qualifier such as “generally positive” or “largely positive” to make explicit the expectation of imperfection, which may be the best that can be hoped for in the near future, at least. There could also be some sort of rubric so that positiveness of assessments is objective, though this will require thoughtful design to avoid the same kinds of issues discussed above.

The role of external assessments in deployment decision-making

Given the nascent ecosystem of external assessment for AI systems, it is important to have clearly defined scopes for such assessments: there are no “one stop shops” available yet, and may never be, since truly mastering all the different considerations would essentially amount to starting a parallel company, and talent is finite. It is more realistic to have different experts assessing different things.

This mindset has several implications for the current draft:

• This requirement doesn’t necessarily make sense as stated: “If Signatories disagree with any recommendations or conclusions reached by external assessors, they commit to specifying their disagreement in the Model Report.” Conclusion may sometimes make sense to respond to but I’m wary of this as a general requirement. In general, assessors should probably not be making net recommendations on deployment, since if they do, they will often be going beyond their expertise in a specific kind of risk. The most capable external assessors today, for example, do not tend to claim expertise beyond a specific area or small number of areas. I don’t want the COP to encourage companies wasting time responding to off-base recommendations and sweeping conclusions based on limited information and expertise, but rather external experts should focus on what they do best – assessing risks in specific areas and stating their conclusions clearly.

• Likewise, the COP says that “a Model Report will describe whether and how external expert assessments informed a decision to place the model on the market, such as through assessment of risks.” Again, external assessors should be aiding assessment of risks, not determining final net decision-making, which involves many factors that assessors will not be privy to.

• The COP says, “Where external assessors submit reports outlining identified risks, testing methodologies, and proposed mitigations to Signatories, Signatories commit to including them in their Model Reports.” Again, given the ecosystem issues described above, these reports may often be of uneven quality and it’d be burdensome to have to continually update, e.g., one’s system cards based on whatever conclusions third parties arrive at. It’s much more important that they be allowed to publish those findings themselves, and that the terms negotiated by companies do not forbid this. It’s also important that regulators have access to any findings upon request. Bundling everything together could dilute the quality, focus, and readability of documentation like system cards and Model Reports.

• KIP 16.2.3 says that “Independent external assessors, under Measure 16.2, receive access to deployed models within 30 days of a formal request, barring exceptional circumstances.” This is vague regarding the type of access in question and the kind of assessors being discussed. If it’s simply black box access, then shouldn’t they have access already by virtue of the product being on the market? If it’s deeper access than that, there will likely be some costs on the company side of providing access (such as staff time) and the terms and intent should be spelled out more here before inclusion in the final COP.

Loophole re: unreported models

If an AI system truly has “systemic risk,” it does not necessarily have to be deployed on the EU market in order to cause harm. It could cause harm to the EU by being deployed commercially elsewhere in a way that still affects the EU, or due to an incident that occurs prior to any external deployment, or because the system is stolen and then misused, or because it’s a non-commercial but deployed system (e.g., for military purposes).

Yet the COP says – based on how the AI Act was designed – that signatories “may choose not to notify the AI Office of a model that meets the classification criteria if they have strong reason to believe they will not put it on the EU market.” This is intended to acknowledge the fact that “some models may be trained solely for internal purposes or as preliminary experiments, with no intention of being placed on the EU market.”

If I understand correctly, this is a big loophole, but also not necessarily fixable from within the COP, since the AI Act is only triggered if things are deployed or will be deployed on the EU market. But I wanted to include this section as a brief reminder that even if we were to achieve perfectly efficient and effective implementation of the AI Act, the problem of mitigating the risks that the AI Act is intended (in part) to address would not be fully achieved.

Perhaps this can be patched in the future, or perhaps the US and other countries will just need to sort out the risks associated with models that aren’t commercially deployed in the EU through other means. I suspect that these unreported models will constitute a growing share of risk over time since there are many reasons to develop powerful (and dangerous) AI systems, not all of which involve deployment on the EU market.

Conclusion

The people writing the Code of Practice are not in an enviable position – they have to consider a zillion stakeholders’ opinions, they have an insane timeline, and they are constrained by the AI Act in terms of which problems they can and can’t solve. I appreciate their efforts even while I point out the remaining room for improvement. 🫡

To sum up, I think compliance with the systemic risk provisions is (mostly) feasible for big companies. But just because it is technically feasible to comply, that doesn’t mean we should just accept AI regulation becoming a box-ticking exercise that ruffles feathers internally for people trying to ship things while not materially increasing the safety of development and deployment practices. So continued innovation will be needed here, and I tried to suggest some ways to do that, including different ways of assessing compliance with the COP as well as quarterly changes to the COP and the systemic risk threshold.

Thinking a bit further out (several months, since things move quickly in AI), there is a fundamental issue here that goes way beyond the AI Act. If the EU keeps ratcheting up the bar for what counts as a model with systemic risk, then yes, they will achieve the goal of focusing on the highest risk systems. But what if more and more models do in fact pose systemic risks over time? Do we focus only on models that pose systemic-er risks?

Ultimately, the world needs to be resilient to misuse and reckless use of models that were once considered to have systemic risks. If that’s not possible, then we’re in big trouble, regardless of what the AI Office does. I am hopeful that concerted investment will get us there, and regulators like the AI Office can over time focus less on absolute thresholds like the ability to hack or generate bioweapons etc., since we will have learned to deal with those, and focus more on relative capabilities. But this is a hypothesis that hasn’t been proven, and I don’t think we should bet the farm on everything being sorted out with innovation.

It’s important that we get the General-Purpose AI Code of Practice right because it’s one of the only “shots on goal” today for building the institutional muscle to do this kind of triaged AI governance, and we don’t yet know if the US will take its own shot in the near future. If you think you could help with this, consider providing feedback on the third draft which is coming soon.

Acknowledgments: Thanks to Dean Ball for helpful comments on earlier versions of this post. Views expressed here are my own.

I’ve covered some of these points in previous blog posts and tweets but wanted to put them all in one place.

For the purpose of this blog post, I define security simply as the inability to be stolen or tampered with, and safety as the absence of accidents, though that’s a bit of a simplification.

Timing disconnect

We are arguably already way behind in terms of security relative to the importance of AI as a technology. Independent experts, drawing on interviews with staff at frontier AI companies, think that such companies are years from being able to defend AI technologies against well-resourced attacks by countries like China and Russia. And that’s if we try hard.

I hope that, as with people saying in 2020 that it’d take many years to make COVID vaccines, the experts are wrong here and we can move faster than current estimates with a concerted effort. But in any case we need to get started ASAP, and I don’t have a very solid basis for my hope.

It’s important to bear in mind that AI systems a few years from now, or even one year from now, will be much more capable. AI companies already have AI systems that are better at competitive coding tasks than all but a few hundred humans, and much more progress is expected soon. I don’t want stuff like that being stolen and abused by whoever happens to good at hacking, espionage, etc.

One calendar year is a long time in “AI years,” so the current situation is just unreasonable at a societal level (which, to be clear, I think most involved would agree with, but it hasn’t been solved yet for the incentive reasons I discuss below).

Interstellar (Paramount Pictures)

Conceptual precedence

In some sense, security takes priority for a technology that can be copied and that is very general-purpose (and thus amenable to all sorts of misuse).

You can’t govern a technology if it won’t sit in one place, and you can’t make that technology controllable if random bad actors are tampering with it, poisoning its data, etc.

Nor can you ensure good safety practices, that may require a lot of care and discussion to get right, if you don’t know who you should even be talking to.

Sometimes it’s good to give everyone access to a technology. But that will not always be a good idea, and we shouldn’t be happy with a situation in which we don’t even have the option of fully protecting any AI IP.

“Don’t Copy that Floppy,” Software Publishers Association

Incentive problems

Unless and until governments, civil society, and/or the public put strong pressure on frontier AI companies to protect their technology rigorously, they will not really pay the full costs of not doing so, and they will have a disincentive to get far ahead of everyone else on security if doing so might slow them down.

Consider theft of American AI IP by Russian or Chinese hackers, for example. A company with lax security may not even find out about this, and if they did, it may not be made public, making it difficult for them to really be penalized for it. And while the costs of investing in security are largely borne by the company, the costs of under-investing are borne by a much larger set of actors, e.g. perhaps the country or world as a whole.

Lastly, acting unilaterally to push super hard on security could slow down research and product progress and give others a leg up in the market. So we should not be expecting this to get solved automatically.

While sorting these incentive issues out is tricky, I think it’s clearly in the national and global interest to do so. While in some cases there will be delays and costs as a result of taking security more seriously, doing so would, overall, help sustain the US’s leadership position in AI and ultimately improve safety and governance outcomes.

(There are incentives to cut corners on safety sometimes, as well as incentives not to do so, which is also true of security. I can’t really do the comparison justice here but the bottom line is that in my view, security is the less well-incentivized one, at least at the standard we should be aiming for.)

Conclusion

Increasing the incentive and technical ability of AI companies to have good security is a very high priority, and in my view even more urgent than safety (though that’s also very important).

While I don’t love the idea of leaving safety until later and in reality we should just be multi-tasking, it’s worth noting that AI safety research can more plausibly be automated at later stages than (some aspects of) security can. AI system N, if safe-ish, can be used to help make AI system N+1 safer through (e.g.) better designing its training process. But building secure chips and datacenters, sorting out security clearances, etc. will inherently take a fair amount of time — more time than the gap between AI model generations. Many parts of security involve atoms (e.g., requiring different chip features and datacenter designs), whereas safety is more of a bits-related problem, and it’s easier to speed up improvements to bits than to atoms.

Security comes first, and should be pursued in a coordinated way so that no one has to put themselves at a disadvantage by making the first move. I hope that this will be one of the highest priorities of the incoming Trump administration.

For further reading on this topic, you might want to check out this long RAND report cited above, or this short version of it, and this report on technical challenges related to security (and other topics) that could benefit from more research.

As discussed in recent posts, I’m quite interested in how companies and countries can demonstrate that the AI systems they’re building are safe, secure, and beneficial, while also respecting important constraints like privacy and intellectual property.

This is a tricky problem, but notably, it’s not unprecedented. Nuclear and chemical facilities are routinely inspected by third parties because people invested a lot in making it work. Financial auditors build confidence daily in the fact that almost all companies in countries with good rule of law aren’t routinely defrauding people.

In this post, I want to outline one idea for how you might go about making AI development and deployment more trustworthy in the near-term, by leveraging human intelligence and institutions, and build a bridge to the longer-term, more automated solution which I think we’ll need eventually.

I’m not sure about the exact details of this idea being right, but hopefully the sketch of one solution below will spark some creative thinking on the overall problem, and maybe there’s something here that can be usefully combined with other ideas.

I won’t go into detail about existing efforts to build trust in AI development and deployment, of which there are many (and many similarities/differences to what I discuss here), but I’ll return to this in a future blog post. In short, there are some nascent efforts in this direction in specific “verticals” (e.g., independent evaluation of dangerous capabilities) but typically they involve much more shallow access than what I describe here.

GAAIO in brief

An organization could be created in order to verify the claims made by companies and countries about AI – and that’s it. It could be called the Global Association of AI Ombudspeople (GAAIO, pronounced “guy-oh”).

Using GAAIO’s services would be totally voluntary, its conclusions would have no force beyond what governments choose to do with them, and it would not suggest claims that it wants companies or countries to make.

To inform GAAIO’s conclusions, which might be public or not depending on the context, GAAIO’s ombudspeople would essentially be given the same access as a normal employee of a company or a government agency.

Keeping the scope and legal status of this organization limited is perhaps the best way to get going, since it would ensure focus and it would not require anyone to sign off on it before getting started. Of course, the stipulation that GAAIO’s services be voluntary means that it would not (on its own) help much with some of the hardest cases involving countries that mistrust one another and especially sensitive use cases like military AI. But I think it could get the ball rolling and establish a practical and theoretical foundation for later steps, and add value on its own.

What sorts of claims and why verify them?

Companies and countries make a lot of claims about the AI that they are developing, deploying, or governing, and these claims vary widely in terms of subjectivity and scope. Sometimes the conclusions may not be a binary yes or no, but it may nevertheless be important for there to be a way of increasing trust in the claims, as part of a larger strategy for AI governance (e.g., regulation of the biggest risks, or multiple countries agreeing to follow certain “rules of the road” and not wanting others to cut corners secretly). Some claims might include, e.g., “we made a good faith effort to red team this system,” “the model we recently declared to government authorities is our most capable model,” “the model we deployed in production is the same one we evaluated and discussed in this system card,” “we don’t have a secret underground datacenter,” etc.

Verifying claims about AI is critical for the same reason that the International Atomic Energy Agency (IAEA), for all its flaws, is critical. For a high-stakes technology like nuclear energy, you can’t just take organizations at their word, at least not in all cases, and an atmosphere of distrust and suspicion can lead to corner-cutting and conflict. So even though it’s difficult sometimes, it is worth it to invest serious time, money, and effort in finding ways to build trust.

Reagan famously said “trust but verify” in the context of arms control with the Soviet Union. I think a similar mindset will ultimately be needed as AI gets more capable (though I’d perhaps rephrase it to “trust through verification,” since in the very hardest cases, like US-China relations, there will be no trust in the beginning).

Doing what doesn’t scale but can work for now

As discussed elsewhere, I think AI will likely play a big part in governing AI, and in a later section I talk about why automation will also be needed. Imagine, e.g. “automated auditors” that are given full access to internal info, spit out a yes/no/maybe answer to a question their human supervisor asked them, and then delete themselves and everything they learned — that’d be awesome and make it much easier to solve some thorny problems in AI policy. I want to help make sure those things ultimately exist but for now, we don’t have them, and AI policy isn’t waiting for them. So what do we have already?

Humans and institutions. While IAEA inspectors have fancy tools, their most important “tools” are their body and brain. These inspectors physically visit facilities, ask questions of staff, use various sensors, and generally bring their expertise to bear on the task at hand.

And those inspectors are part of a larger institution that has a reputation built over decades, that has explicit values and processes around protection of sensitive information, that has training and apprenticeship programs, and that draws on research insights from the IAEA’s own research facilities as well as the scientific community more generally. There’s nothing stopping us from building a team of AI ombudspeople to do just that in an AI context, with this team being drawn from various countries around the world, and “deploying” them in a matter of months (again, at the invitation of the organization/country in question).

There would be a need for some trial and error and working up from lower to higher stakes cases, but that was and is also true in the nuclear context, with lab-scale pilots, large-scale experiments like the Joint Verification Experiment and the Black Sea Experiment, and ultimately the full-blown deployment of new processes and technologies at global scale by the IAEA, which does hundreds of inspections a year.

How this might work

Ombudspeople from GAAIO would effectively be “employees for a day/week/month” with a very specific role (assessing specific claims they were asked to assess by the company or country in question), or they would do a long-term residency and assess the organization in a more holistic and rolling fashion. They would review documents, interview staff, and run experiments as needed, and would need the skills to make sense of all of this information.

Since AI is a general-purpose technology, it’s impossible to exhaustively illustrate all the ways that this could work in practice, but let me give one example.

Jane Smith is an American research engineer that recently left a frontier AI company. She applies to become an ombudsperson at GAAIO, since she wants to do her part to reduce the risk of a “race to the bottom” on AI, and she starts a few months later after an interview/screening process and some training. On her first assignment, she is assigned to a Singaporean AI company that wants a larger profile in the AI policy debate, so they decide to be one of GAAIO’s first “guinea pigs” and filled out a form on GAAIO’s website. Jane spends a month at this company, the first week of which is spent going through the normal onboarding process, including getting a company laptop, meeting people on various teams, and learning the company’s software stack.

Jane, in collaboration with support staff at GAAIO and executives at the company, negotiates the terms for the output of this visit. Specifically, she is asked to publicly assess the claims made in the company’s latest system card about deception occurring in the language model’s “chain of thought.” The company had claimed in this system card that they rigorously studied the issue and found that, while sometimes there was a difference between what the model said to itself while thinking and what it said to the user afterwards, these discrepancies were always explainable in some reasonable way, such as the model complying with its own behavioral policies as it understood them (e.g., not wanting to hurt a user’s feelings). Some AI safety researchers were not convinced by this few sentence explanation, and posted on social media about how the company had not provided sufficient evidence of this claim — leading the company to reach out to GAAIO.

In her remaining three weeks at the company, Jane interviews everyone involved in the deception research summarized in the system card, as well as staff that were indirectly involved (e.g., she speaks with people who make product deployment decisions and people who allocate compute to different research projects). Overall, she is satisfied with what seemed to be an honest intention at the company to do well in this area, and she does not see any indication of the staff trying to mislead her, and she is impressed with many aspects of the experiments they did. It also seemed like they had plenty of resources and support to do their work during the critical period leading up to launch.

But Jane also quickly learns that the skillset of the staff did not include mechanistic interpretability research, so it is inherently hard for them to really, truly understand what is going on in the language model’s “head.” It is not 100% clear, based on the input-output behavior of the model alone, whether the model’s stated reasons for certain outputs (e.g., adhering to a policy it was trained to follow) are really the real reasons — e.g., perhaps the model also was somehow considering factors like the possibility of getting shut down in the future if it said one thing versus another, but it had learned not to talk about these things. Fortunately, Jane has a background in this area, and she worked with some engineers at the company to integrate some open source interpretability software with the company’s own software stack, and do some initial experiments. She isn’t able to totally resolve the issue in the short time available, but she helps at least understand what the gaps are in mor edetail.

In the month after leaving the company, Jane distills her scattered notes into a report and adds references to related research. In collaboration with the company, GAAIO publishes a blog post summarizing her findings. The blog post was sent to the company beforehand for feedback, though this feedback only involves making sure GAAIO is following the agreed-upon terms on what can and can’t be disclosed, and the company providing supplemental information that GAAIO can consider including. In short, the report says that the claims made in the system card appeared to have been the genuine views of the company, and that they did indeed do a fairly in-depth analysis of the topic, by the prevailing standards of the industry at the time. However, she also notes the remaining limitations of this work and some ways in which these claims could be made more robust in the future. Her findings help to inspire an uptick in research at the intersection of mechanistic interpretability and chain of thought by researchers at a few organizations. The report also includes some process-related lessons learned from this early pilot, in order to inform other companies’ preparation for possible ombudsperson visits as well as policy researchers and policymakers thinking about analogous ideas.

Some challenges and solutions

Doesn’t this constitute a huge breach of confidentiality on the part of the host organization? The ombudspeople would access sensitive information, which is why you wouldn’t go from 0 to 60 overnight, (e.g., having PLA officers hang out at NSA all day and vice versa), and may never reach 60. But we can do better than what we have now. Experience has shown in other contexts that where there is a desire for verification of claims, processes and technologies can be developed that limit the downside risk to the host organization, and even give big benefits to the host organization by giving their claims credibility – it just takes time, money, and effort.

While there are many permutations on this idea, the simplest version is just giving access that is equivalent to a normal employee — making the risk roughly equivalent to a single employee being an insider threat. But I think in practice, the risk would be lower than that given the reputational implications for GAAIO if they screw up, and the fact that the terms of access would likely be discussed and agreed on in much more depth than typical employee onboarding.

Furthermore, GAAIO could consider various other ways to further reduce real and perceived risks of their ombudspeople abusing their access — for example, assignment of specific ombudspeople to specific host organizations may be done in a random fashion so that no one can meddle with the process in order to engineer a certain outcome (e.g., spying on a certain country). There could also be some sort of country-based considerations to further build trust, e.g., perhaps only candidates from countries that do not have adversarial relationships with the host country might be eligible for a given assignment, or all employees would have to commit publicly to never joining a competing AI effort afterwards.

There would probably need to be a legal agreement in place on what the ombudspeople can say about their experience after above and beyond the topic they were asked to investigate, which may be shared openly or privately with another party (e.g., a regulator) depending on the negotiated terms.

Over time, as in other cases, a standard formula would develop, but it would be important to make sure that things don’t get too over-engineered: it will be important that the ombudsperson be encouraged to be creative and adaptive in how they go about their work in particular contexts, given the huge variety of possible claims, organizations, and AI systems. And if they encounter artificial obstacles (ie., they aren’t given the same access as an employee would normally be given), then just as with IAEA inspectors, this fact should be noted as a caveat in any assessments they produce.

Scalability for the longer term

The reasons why the approach above might work at all also helps to explain why it won’t last forever.

Three facts make this approach potentially tractable today.

1. There is a relatively small number of AI systems that (currently) pose or will in the near future pose catastrophic risks, since building such systems is (currently) expensive. Estimates vary but illustratively, you might imagine single digits to dozens or hundreds of such systems — comparable to the number of nuclear facilities handled by the IAEA.

2. The internal environment at organizations building these technologies is relatively open and permissive. There are some technical efforts to prevent outright theft of some IP like model weights (albeit not enough). But generally speaking, information flows freely, so “employee-level access” means fairly deep and wide access to internal systems, knowledge, activities, and plans — enough to do the job of an ombudsperson at least as effectively as a given employee could verify their own employer’s claims.

3. The pace of AI progress, while rapid, is still largely human-driven and human-understandable, and systems are not too much smarter than people (and therefore well-qualified ombudspeople) in any particular domain.

At least the first and third reasons won’t last much longer. It’s hard to say what will happen in terms of internal transparency, but it’s very clear that over time, AI capabilities will get cheaper to build (so for any given level of risk, there will be more such systems), more AI research will get automated, and the peak of AI capabilities will increase. This will challenge the ability of GAAIO or similar organizations to do their job at a scale commensurate with the “surface area” of AI governance.

Ultimately, the solution will involve GAAIO and other institutions adopting AI heavily in their work, such as with the automated auditor idea mentioned above. Humans will likely still be significantly involved in decision-making about AI but they will draw heavily on the input of smarter systems than themselves, and their role will evolve as a result. And society will need to dramatically increase its investment in resilience to risks from AI in order to ensure that it can weather the shocks from all of those dangerous AIs, some of which will be misused deliberately or developed and deployed recklessly.

Governance attention will and should continue to focus on the most capable systems, which will likely be the most compute-intensive and smaller in number than the less capable systems, but the overall “intelligence waterline” will be much higher than it is today, and we will not be able to rely on humans manually reviewing every (e.g.) superhuman hacking system.

For now, humans are in the driver seat of AI capabilities, safety, and policy, and we should make the most of this period while preparing for the next. The automated systems we will need for the next phase may be built with today’s more human-centric and slow-paced approach. We have an opportunity to shape AI today and the initial conditions of AI tomorrow, and should put more human brainpower on the verification while planning ahead for, and building, scalable solutions.

Conclusion

Making AI development and deployment trustworthy is at the heart of many AI policy challenges, domestically and internationally. So we need to develop and try out various ideas that could help achieve this, and I hope the GAAIO concept sparks some related thinking.

At a domestic level, and for domestic regulations with global ramifications like the EU AI Act, the “employee for a week” and long-term residency models described above could potentially help to address the growing limitations of the current regulatory paradigm focused on tracking pretraining FLOPs. At an international level, something like GAAIO —combined with a sensible approach to compute governance, and commitments/claims made via other channels — could serve as an 80/20 version of the “IAEA for AI” concept.

I would love to see more people debating, fleshing out, and piloting ideas like GAAIO. As always, feel free to reach out if you’re interested in connecting about this or other topics I’ve written about lately.

Acknowledgments: Thanks to Larissa Schiavo for helpful comments.

AI policy decisions made in the next few months will shape how that AI is governed. The security and safety measures in place for safeguarding that AI will be among the most important in history. Key upcoming milestones include the first acts of the Trump administration, the first acts of the next US congress, the UK AI bill, and the EU General-Purpose AI Code of Practice.

If there are ways that you can help improve the governance of AI in these and other countries, you should be doing it now or in the next few months, not planning for ways to have an impact several years from now.

The announcement of o3 today makes clear that superhuman coding and math are coming much sooner than many expected, and we have barely begun to think through or prepare for the implications of this (see this thread) – let alone the implications of superhuman legal reasoning, medical reasoning, etc. or the eventual availability of automated employees that can quickly learn to perform nearly any job doable on a computer.

There is no secret insight that frontier AI companies have which explains why people who work there are so bullish about AI capabilities improving rapidly in the next few years. The evidence is now all in the open. It may be harder for outsiders to fully process this truth without living it day in and day out, as frontier company employees do, but you have to try anyway, since everyone’s future depends on a shared understanding of this new reality.

It is difficult to conclusively demonstrate any of these conclusions one way or the other, so I don’t have an airtight argument, and I expect debate to continue through and beyond the point of cross-domain superhuman AI. But I want to share the resources, intuitions, and arguments I find personally compelling in the hopes of nudging the conversation forward a tiny bit.

This blog post is intended as a starter kit for what some call “feeling the AGI,” which I defined previously as:

• Refusing to forget how wild it is that AI capabilities are what they are

• Recognizing that there is much further to go, and no obvious "human-level" ceiling

• Taking seriously one's moral obligation to shape the outcomes of AGI as positively as one can

(I will focus on the first two since the third follows naturally from agreement on the first two and is less contested, though of course what specifically you can do about it depends on your personal situation.)

How far we’ve come and how it happened

It has not always been the case that AI systems could understand and generate language fluently – even just for chit chat, let alone for solving complex problems in physics, biology, economics, law, medicine, etc. Likewise for image understanding and generation, audio understanding and generation, etc.

This all happened because some companies (building on ideas from academia) bet big on scaling up deep learning, i.e. making a big artificial neural network (basically just a bunch of numbers that serve as “knobs” to fiddle with), and then tweaking those knobs a little bit each time it gets something right or wrong.

Language models in particular first read a bunch of text from the Internet (tweaking their knobs in order to get better and better at generating “text that looks like the Internet”), and then they get feedback from humans (or, increasingly, from AI) on how well they’re doing at solving real tasks (allowing more tweaking of the knobs based on experience). In the process, they become useful general-purpose assistants.

It turns out that learning to mimic the Internet teaches you a ton about grammar, syntax, facts, writing style, humor, reasoning, etc., and that with enough trial and error, it’s possible for AI systems to outperform humans at any well-defined task. This line of thinking is what led to o3, and there is much more to come as discussed below.

The fact that this all works so well — and so much more easily and quickly than many expected — is easily one of the biggest and most important discoveries in human history, and still not fully appreciated.

Here are some videos that explain how we got here, and some other key things to know about the current trajectory of AI. I draw especially from Ilya Sutskever since he is a very clear communicator and one of the leading authorities on AI.

(next-token prediction is the thing I mentioned above about learning to mimic the Internet — language models make zillions of guesses about what comes next in a sequence of words, or tokens — which are parts of words — and in the process they learn a lot of useful skills.)

Here are some other long reads on related topics. As with the videos, I don’t endorse all of the claims in all of these references, but in the aggregate I hope they give you some 80/20 version of what people at the leading companies know and believe, though I also think that regularly using AI systems yourself (particularly on really hard questions) is critical in order to build up an intuition for what AI is capable of at a given time, and how that is changing rapidly over time.

There is no wall and there is no ceiling

There is a lot of “gas left in the tank” of AI’s social impacts even without further improvements in capabilities — but those improvements are coming.

The most capable models such as o3 have not been deployed yet, and there has barely been time for society to process the impacts of recent deployments like Claude 3.5 Sonnet (from Anthropic) and o1 (from OpenAI). Gemini 2.0 Flash (from Google) is quite impressive but we don’t yet know what the big versions of that model family will look like, and likewise for Claude 3.5 Opus (the biggest version of that family).

And that’s all just in the very immediate term.

A bit further out (again, months to a small number of years at the very most), we will see a true scaling up of reinforcement learning on language models, the integration of this paradigm with “computer-using agents,” much bigger datacenters, and the increasing automation of AI research and engineering itself via AI. Again, this doesn’t require any major new paradigms, just hard work by thousands of people over time, and countless experiments, bug fixes, etc. along the way. There is no shortage of ideas for turning computing power into intelligence, and while some may be a bit easier to frame as reinforcement learning problems, nothing doable on a computer is safe

Here are some videos that give a sense of what I mean.

Note that it is not just researchers but also the CEOs of these companies who are saying that this rate of progress will continue (or accelerate). I know some people think that this is hype, but please, please trust me — it’s not.

We will not run out of ideas, chips, or energy unless there’s a war over AI or some catastrophic incident that causes a dramatic government crackdown on AI. By default we maybe would have run out of energy but it seems like the Trump administration and Congress are going to make sure that doesn’t happen. We’re much more likely to run out of time to prepare.

Where to go from here

I’ve written a bit about what I think the Trump administration should do on AI, and I would add that I’ve recently come to the conclusion that security should be the top priority. If you can’t control where AI weights go (i.e., prevent leaking or theft), you can’t govern the technology. It’s that simple. So we need a large-scale public-private partnership to build secure chips and datacenters. See related discussion here.

There are many other things that should be done, including making sure that the General-Purpose AI Code of Practice is well-designed, and that Congress doesn’t forget about this bipartisan report they just put out. Congress should also authorize substantial funding for the US AI Safety Institute and the Bureau of Industry and Security, etc.

I could go on, but my larger point is not that I have the perfect blueprint, or that there’s no need for further thinking about getting the details mostly right. There should be some fundamental thinking on AI policy questions happening — but only about the later stages of AI (i.e., after we already have superhuman capabilities), not about the steps right in front of us like getting solid security in place before those capabilities exist. We don’t have time for 10,000 more papers and 100 more blue-ribbon commissions.

By the end of 2025, if not earlier, we need comprehensive AI regulation in the US, a dramatic improvement in government capacity on AI (i.e. number of AI engineers/scientists with security clearance, etc.), dramatic improvements in safety culture and processes at companies, significant progress in international dialogue on rules of the road for military AI, and much more.

See here for AI policy career advice, and as always, feel free to reach out via this form or using the contact info on my website.