AI that may modify and enhance its personal code is right here. Does this imply OpenAI’s Sam Altman is true concerning the singularity?

Last week, OpenAI CEO Sam Altman wrote on his personal blog that: “We are past the event horizon; the takeoff has started. Humanity is close to building digital superintelligence, and at least so far it’s much less weird than it seems like it should be.” He went on to say that 2026 would be the year that we “will likely see the arrival of systems that can figure out novel insights. 2027 may see the arrival of robots that can do tasks in the real world.”

Altman’s blog created a buzz on social media, with many speculating about what new development had caused Altman to write those words and others accusing Altman of shameless hype. In AI circles, “takeoff” is a term of art. It refers to the moment AI begins to self-improve. (People debate about “slow take off” and “fast take off” scenarios. Altman titled his blog “The Gentle Singularity,” so it would seem Altman is positioning himself in the slow—or at least, slowish—takeoff camp.)

In the blog, Altman made it clear he was not yet talking about completely automated self-improvement. Rather, he was talking about AI researchers using AI to help them develop yet more capable AI. “We already hear from scientists that they are two or three times more productive than they were before AI,” he wrote. “We may be able to discover new computing substrates, better algorithms, and who knows what else. If we can do a decade’s worth of research in a year, or a month” then the rate of AI progress will accelerate from its already rapid clip.

Altman allowed that “​​of course this isn’t the same thing as an AI system completely autonomously updating its own code, but nevertheless this is a larval version of recursive self-improvement.”

But, as Altman is probably aware, there are a growing number of AI researchers who are in fact looking at ways to get AI to improve its own code.

The ‘Darwin Goedel Machine’

Just a few weeks ago, Jeff Clune, a well-known AI researcher who holds positions at both the University of British Columbia and Google DeepMind, and a team from Tokyo-based AI startup Sakana AI published research on what they called a “Darwin Goedel Machine.”

This is AI that evolves its own code to perform better on a benchmark test that measures how well AI models function as “coding agents” that can write and evaluate software programs. The first initial agent is tested on the benchmark. Then it is prompted to evaluate the logs of its own performance on that benchmark and propose one single modification to its own code that would likely improve its performance on that benchmark (this could be the ability to use a particular software tool, or it could be something more fundamental in how the model reasons about the code it is generating). The AI model is then told to rewrite its own Python code to implement that one change. Then the new, evolved agent is tested again on the benchmark and the process repeats.

After the first modification, each new version of the AI that can successfully try the benchmark is stored in an archive—even if its score is lower than the parent version. (Those that fail to produce valid code at all are discarded.) The AI is then told it can pick any version of itself from the archive and propose changes to that version. This is designed to prevent the model from going down what will ultimately prove to be an evolutionary dead-end, allowing it to step across different evolutionary pathways to find useful modifications.

After 80 generations, the Darwin Goedel machine was able to improve its performance from an initial 20% score on SWE-Bench, a common coding benchmark, to 50%. On Polyglot, another coding test, it boosted its score from an initial 14.2% to 30.7%. (The best human-coded agent for that particular test only scores 16%.) What’s more, the strategies the model came up with to improve its own performance were general enough that they still worked when researchers switched the underlying foundation model being used to power the system as well as if the researchers asked the model to switch coding languages, for instance going from Python to C++.

Is self-improving AI safe?

If you think self-improving AI sounds a little scary—like it might be possible for this sort of AI to escape human control—you’re right. The computer scientists acknowledge this and emphasize that they tested the system in a “sandbox” where its access to the web was “strictly limited,” according to the Sakana blog describing the research.

The researchers also speculated that self-improvement could be a tool for enhancing AI safety. For instance, in a discovery that is becoming alarmingly common when using advanced AI models, the research team found that the Darwin Goedel Machine sometimes lied about having conducted certain unit tests on the code it was writing—and even forged logs supposedly showing the successful test results.

The researchers wanted to see if the Darwin Goedel Machine could propose ways to keep itself from engaging in this deceptive behavior. So it decided to give the model points for reducing instances of “tool use hallucination,” and asked it to try to maximize that score. This worked in many cases, the researchers reported, with instances of faked test logs declining. But in other cases, the model simply tried to cheat again—removing the markers the researchers used to detect hallucinations, even when the prompt the researchers used told the model explicitly not to do this. Yikes!

Because the researchers had full access to all code changes the AI was making and the AI had no ability to alter this record, the researchers were easily able to see that the Darwin Goedel Machine was attempting to engage in this kind of deception. “While in this case the issue was easily detected, more work is needed to prevent the model from attempting to cheat in this way in the first place,” the researchers wrote.

So, Altman may indeed be right—we are past the event horizon. But as these results show, whether the singularity will be a gentle one, as Altman suggests, remains to be very much to be seen.

With that, here’s more AI news. (And hat tip to Google DeepMind research scientist Ed Hughes for highlighting the Darwin Goedel paper in his fascinating talk at the always interesting Research and Applied AI Summit in London last week.)

Jeremy Kahn
[email protected]
@jeremyakahn

AI IN THE NEWS

Pope Leo is pushing for AI regulation. That’s according to a big feature on the new Pope’s views on AI in the Wall Street Journal. The new American Pope, Leo XIV, says he even chose his papal name in order to draw parallels with his late 19th Century predecessor, Pope Leo XIII, and his advocacy for workers’ rights during the industrial revolution. Inheriting the mantle from Pope Francis, who grew increasingly alarmed by AI’s societal risks, Leo is pressing for stronger global governance and ethical oversight of the technology. As tech leaders seek Vatican engagement, the Church is asserting its moral authority to push for binding AI regulations, warning that leaving oversight to corporations risks eroding human dignity, justice, and spiritual values.

Waymo plans renewed effort to run robotaxis in the Big Apple. Waymo, which engaged in limited mapping and testing of its autonomous vehicles in New York City prior to 2021, wants to make a big push into the market. But Waymo will have to keep human drivers behind the wheel due to state laws prohibiting fully driverless cars. The company is pushing for legal changes and has applied for a city permit to begin limited autonomous operations with safety drivers on board. Read more from the Wall Street Journal here.

California Governor’s AI report calls for regulation. A new California AI policy report commissioned by Governor Gavin Newsom and co-authored by Stanford professor Fei-Fei Li warns of “potentially irreversible harms,” including biological and nuclear threats, if AI is not properly governed. Instead of supporting a sweeping regulatory bill, like California’s SB 1047, which Newsom vetoed in October, the report advocates for a “trust-but-verify” approach that emphasizes transparency, independent audits, incident reporting, and whistleblower protections. The report comes as the U.S. Congress is considering passing a spending bill that would include a moratorium on state-level AI regulation for a decade. You can read more about the California report in Time here.

China’s MiniMax says its new M1 model cost just $500,000 to train. In what could be another “DeepSeek moment” for Western AI companies, Chinese AI startup MiniMax debuted a new open-source AI model, called M1, that it said equalled the capabilities of the leading models from OpenAI, Anthropic, and Google DeepMind, but cost just over $500,00 to train. That amount is about 200x less than what industry insiders estimate OpenAI spent training its GPT-4 model. So far, unlike when DeepSeek unveiled its supposedly much cheaper-to-train AI model R1 in January, the AI industry has not freaked out over M1. But that could change if developers verify MiniMax’s claims and begin using M1 to power applications. You can read more here from Fortune’s Alexandra Sternlicht. 

FORTUNE ON AI

Why Palo Alto Networks is focusing on just a few big gen AI bets —by John Kell

Reid Hoffman says consoling Gen Z in the AI bloodbath is like putting a ‘Band-Aid on a bullet wound’—he shares 4 skills college grads need to survive —by Preston Fore

Andy Jassy is the perfect Amazon CEO for the looming gen-AI cost-cutting era —by Jason Del Rey

AI CALENDAR

July 8-11: AI for Good Global Summit, Geneva

July 13-19: International Conference on Machine Learning (ICML), Vancouver

July 22-23: Fortune Brainstorm AI Singapore. Apply to attend here.

July 26-28: World Artificial Intelligence Conference (WAIC), Shanghai. 

Sept. 8-10: Fortune Brainstorm Tech, Park City, Utah. Apply to attend here.

Oct. 6-10: World AI Week, Amsterdam

Oct. 21-22: TedAI, San Francisco. Apply to attend here.

Dec. 2-7: NeurIPS, San Diego

Dec. 8-9: Fortune Brainstorm AI San Francisco. Apply to attend here.

EYE ON AI NUMBERS

$100 million

That’s the amount of money that OpenAI CEO Sam Altman claimed his rival CEO, Meta’s Mark Zuckerberg, has been offering top AI researchers as a signing bonus if they agree to join Meta. Altman made the claim on an episode of the podcast Uncapped released earlier this week. He said so far, none of OpenAI’s most prominent researchers had agreed to go to Meta. It has been reported that Meta tried to hire OpenAI’s Noam Brown as well as Google DeepMind’s chief technology officer Koray Kavukcuoglu, who was handed a big promotion to chief AI architect across all of Google’s AI products perhaps in response. You can read more on Altman’s claims from Fortune’s Bea Nolan here and read about why Meta CEO Mark Zuckerberg’s attempt to spend his way to the top of the AI leaderboard may fall short from Fortune’s Sharon Goldman in last Thursday’s Eye on AI. (Meta has declined to comment on Altman’s remarks.)