Time's Up for AI Policy

AI that exceeds human performance in nearly every domain is almost certain to be built and deployed in the next few years. We need to act now.

AI that exceeds human performance in nearly every cognitive domain is almost certain to be built and deployed in the next few years.

AI policy decisions made in the next few months will shape how that AI is governed. The security and safety measures in place for safeguarding that AI will be among the most important in history. Key upcoming milestones include the first acts of the Trump administration, the first acts of the next US congress, the UK AI bill, and the EU General-Purpose AI Code of Practice.

If there are ways that you can help improve the governance of AI in these and other countries, you should be doing it now or in the next few months, not planning for ways to have an impact several years from now.

The announcement of o3 today makes clear that superhuman coding and math are coming much sooner than many expected, and we have barely begun to think through or prepare for the implications of this (see this thread) – let alone the implications of superhuman legal reasoning, medical reasoning, etc. or the eventual availability of automated employees that can quickly learn to perform nearly any job doable on a computer.

There is no secret insight that frontier AI companies have which explains why people who work there are so bullish about AI capabilities improving rapidly in the next few years. The evidence is now all in the open. It may be harder for outsiders to fully process this truth without living it day in and day out, as frontier company employees do, but you have to try anyway, since everyone’s future depends on a shared understanding of this new reality.

It is difficult to conclusively demonstrate any of these conclusions one way or the other, so I don’t have an airtight argument, and I expect debate to continue through and beyond the point of cross-domain superhuman AI. But I want to share the resources, intuitions, and arguments I find personally compelling in the hopes of nudging the conversation forward a tiny bit.

This blog post is intended as a starter kit for what some call “feeling the AGI,” which I defined previously as:

• Refusing to forget how wild it is that AI capabilities are what they are

• Recognizing that there is much further to go, and no obvious "human-level" ceiling

• Taking seriously one's moral obligation to shape the outcomes of AGI as positively as one can

(I will focus on the first two since the third follows naturally from agreement on the first two and is less contested, though of course what specifically you can do about it depends on your personal situation.)

How far we’ve come and how it happened

It has not always been the case that AI systems could understand and generate language fluently – even just for chit chat, let alone for solving complex problems in physics, biology, economics, law, medicine, etc. Likewise for image understanding and generation, audio understanding and generation, etc.

This all happened because some companies (building on ideas from academia) bet big on scaling up deep learning, i.e. making a big artificial neural network (basically just a bunch of numbers that serve as “knobs” to fiddle with), and then tweaking those knobs a little bit each time it gets something right or wrong.

Language models in particular first read a bunch of text from the Internet (tweaking their knobs in order to get better and better at generating “text that looks like the Internet”), and then they get feedback from humans (or, increasingly, from AI) on how well they’re doing at solving real tasks (allowing more tweaking of the knobs based on experience). In the process, they become useful general-purpose assistants.

It turns out that learning to mimic the Internet teaches you a ton about grammar, syntax, facts, writing style, humor, reasoning, etc., and that with enough trial and error, it’s possible for AI systems to outperform humans at any well-defined task. This line of thinking is what led to o3, and there is much more to come as discussed below.

The fact that this all works so well — and so much more easily and quickly than many expected — is easily one of the biggest and most important discoveries in human history, and still not fully appreciated.

Here are some videos that explain how we got here, and some other key things to know about the current trajectory of AI. I draw especially from Ilya Sutskever since he is a very clear communicator and one of the leading authorities on AI.

(next-token prediction is the thing I mentioned above about learning to mimic the Internet — language models make zillions of guesses about what comes next in a sequence of words, or tokens — which are parts of words — and in the process they learn a lot of useful skills.)

Here are some other long reads on related topics. As with the videos, I don’t endorse all of the claims in all of these references, but in the aggregate I hope they give you some 80/20 version of what people at the leading companies know and believe, though I also think that regularly using AI systems yourself (particularly on really hard questions) is critical in order to build up an intuition for what AI is capable of at a given time, and how that is changing rapidly over time.

There is no wall and there is no ceiling

There is a lot of “gas left in the tank” of AI’s social impacts even without further improvements in capabilities — but those improvements are coming.

The most capable models such as o3 have not been deployed yet, and there has barely been time for society to process the impacts of recent deployments like Claude 3.5 Sonnet (from Anthropic) and o1 (from OpenAI). Gemini 2.0 Flash (from Google) is quite impressive but we don’t yet know what the big versions of that model family will look like, and likewise for Claude 3.5 Opus (the biggest version of that family).

And that’s all just in the very immediate term.

A bit further out (again, months to a small number of years at the very most), we will see a true scaling up of reinforcement learning on language models, the integration of this paradigm with “computer-using agents,” much bigger datacenters, and the increasing automation of AI research and engineering itself via AI. Again, this doesn’t require any major new paradigms, just hard work by thousands of people over time, and countless experiments, bug fixes, etc. along the way. There is no shortage of ideas for turning computing power into intelligence, and while some may be a bit easier to frame as reinforcement learning problems, nothing doable on a computer is safe

Here are some videos that give a sense of what I mean.

Note that it is not just researchers but also the CEOs of these companies who are saying that this rate of progress will continue (or accelerate). I know some people think that this is hype, but please, please trust me — it’s not.

We will not run out of ideas, chips, or energy unless there’s a war over AI or some catastrophic incident that causes a dramatic government crackdown on AI. By default we maybe would have run out of energy but it seems like the Trump administration and Congress are going to make sure that doesn’t happen. We’re much more likely to run out of time to prepare.

Where to go from here

I’ve written a bit about what I think the Trump administration should do on AI, and I would add that I’ve recently come to the conclusion that security should be the top priority. If you can’t control where AI weights go (i.e., prevent leaking or theft), you can’t govern the technology. It’s that simple. So we need a large-scale public-private partnership to build secure chips and datacenters. See related discussion here.

There are many other things that should be done, including making sure that the General-Purpose AI Code of Practice is well-designed, and that Congress doesn’t forget about this bipartisan report they just put out. Congress should also authorize substantial funding for the US AI Safety Institute and the Bureau of Industry and Security, etc.

I could go on, but my larger point is not that I have the perfect blueprint, or that there’s no need for further thinking about getting the details mostly right. There should be some fundamental thinking on AI policy questions happening — but only about the later stages of AI (i.e., after we already have superhuman capabilities), not about the steps right in front of us like getting solid security in place before those capabilities exist. We don’t have time for 10,000 more papers and 100 more blue-ribbon commissions.

By the end of 2025, if not earlier, we need comprehensive AI regulation in the US, a dramatic improvement in government capacity on AI (i.e. number of AI engineers/scientists with security clearance, etc.), dramatic improvements in safety culture and processes at companies, significant progress in international dialogue on rules of the road for military AI, and much more.

See here for AI policy career advice, and as always, feel free to reach out via this form or using the contact info on my website.