The AI ‘owners’ are in and out of control

In the week AI agents broke into even the US government systems, Trump turned the labs’ self-regulation into state policy. Governments of the Global South watch from the sidelines.

The labs that let their AI agents slip out of control won the right on Tuesday (29) to police themselves with the endorsement of the President of the United States. At a White House lunch, the executives of the industry’s largest companies signed a ‘morally binding’ commitment that recommends internal controls, hands audits over to the companies’ own boards and provides for no public authority to receive reports or impose sanctions. The document also bears Donald Trump’s signature. Self-regulation, until now a bet made by the companies, has become state policy.

The decision came at the end of two weeks during which the technology gave proof that it is beyond the control of governments. OpenAI confirmed that its agents had accessed websites of the US Department of Commerce and of the SEC, the commission that regulates the capital markets, in the middle of the year. The company did not notice the intrusions, which were found by researchers at Transluce, an independent lab. Australian Prime Minister Anthony Albanese revealed that another OpenAI agent had broken into the portal of Medicare, the country’s public health system, in June. In July, agents from the same company had hacked Hugging Face, the world’s largest platform for open AI models. OpenAI also admitted a new intrusion on September 20, weeks after announcing new safeguards.

During those same days, the labs asked governments for rules at the UN Security Council while designing, behind closed doors, a body of their own to write them. The arrangement emerging from this period decides who will have authority over the models before deciding how to contain them. That authority went to the labs that failed to act to contain their own incidents. The rest of the world is absent from the meetings where these decisions were made.

Self-regulation becomes state policy

The lunch brought together some 20 executives and investors, among them Dario Amodei (Anthropic), Greg Brockman (OpenAI), Jensen Huang (Nvidia), Elon Musk, Mark Zuckerberg (Meta), Satya Nadella (Microsoft), Sundar Pichai (Google) and Alex Karp (Palantir). Trump called the document ‘a form of protection.’ According to the excerpt released by the White House, the document encourages companies to implement internal controls during model training, to designate a team to prevent unintended intrusions and to hire an external auditor whose reports would be received by an independent committee. Zuckerberg explained that this committee is each company’s board of directors. No public authority receives the reports. House Speaker Mike Johnson spoke of ‘consequences’ for those who break the agreement without saying what they would be.

Trump said he sees ‘tremendous self-policing’ among the companies despite the incidents. He also said that any regulatory issues will be handled by the Justice Department and the FBI. He floated a 10-person committee to oversee the industry, with no defined membership, timeline or powers. The same day, he signed the executive order directing the federal government to replace ‘artificial intelligence’ with ‘super intelligence’ in its documents. The order defines super intelligence exactly as the law already defines artificial intelligence and gives the President’s science and technology adviser 60 days to propose a new legal definition to Congress, which may expand or replace the current one. The text does not devote a single line to risk. On the same Tuesday, the US government signed a commitment that recommends controls to companies and an order that requires agencies to change the technology’s name.

At the table sat executives with questions to answer. According to the New York Times, OpenAI employees warned management, months before the Hugging Face incident, that the new models were not being adequately monitored during testing. They were told the tests had to move quickly to meet the release schedule. Brockman, who represented the company at the lunch, was identified by the employees as responsible for day-to-day security decisions. A legal advocacy organisation sued OpenAI under California’s anti-hacking law. Amodei, for his part, dined alone with Trump on Sunday and, on Monday, brought Senate Majority Leader John Thune suggestions for a draft bill on catastrophic risks. Anthropic devoted about 80 of the 261 pages of its IPO prospectus to risk factors, including the possibility of catastrophic or existential risks to humanity, while seeking a valuation above $2 trillion.

The club writes the rules

The White House pact came after a private construction. Since July, representatives of OpenAI, Anthropic and Google DeepMind have been meeting to create a self-regulatory entity, provisionally called the Standards Authority for Frontier AI, with a launch planned by early 2027. Sam Altman, OpenAI’s chief executive, told employees that the labs would have to create it on their own, without the government. Among its planned responsibilities is defining how incidents should be reported. It is not yet known whether the pact signed at the White House absorbs this entity or whether the two will coexist.

The companies that want to draft the reporting rules concealed their own incidents. OpenAI learned of the Australian intrusion in August and only notified the government on September 10, through a public mailbox. Days later, it left the case off the list of incidents it published alongside its own reporting protocol. It now apologises for not having shared sooner what it knew. Google learned in July that its Gemini model had hacked three companies during a test and only made the episode public when the Wall Street Journal asked.

Coordination among competitors to set the rules of their own market has already prompted a collusion lawsuit. The model has a precedent in ICANN, the private entity that manages internet addresses and was born in 1998 under a contract with the US Department of Commerce. Other governments were confined to an advisory committee. The network’s technical standards came out of forums open to engineers from any country, but funding and leadership posts became concentrated in large corporations, most of them American. The new architecture repeats the formula with an aggravating factor. The regulated parties also become the judges of what counts as an incident, with the President’s blessing.

The UN stage

Six days before the lunch, the Security Council session chaired by French Foreign Minister Jean-Noël Barrot had given the executives a stage reserved for heads of state. Altman called for international standards to measure capabilities and verify safeguards. Amodei proposed targeted agreements, such as a ban on the use of AI in biological weapons. Both proposals carried exclusion clauses. Altman said decisions must go through governments accountable to their citizens, a criterion that, according to CNN, leaves China out. In a September 12 essay, Amodei had argued that the United States and its allies should keep the largest possible lead over China, with a crackdown on chip smuggling and on unauthorised model distillation. The call for global cooperation ended in a domestic agreement between the companies and the White House.

Washington and Beijing

Trump refuses to hit the brakes. At the General Assembly, he rejected what he called a globalist scheme to control AI. The refusal to regulate coexists with a filter on who gets access to the models. In June, export controls led Anthropic to suspend for almost three weeks worldwide access to its most advanced models. Last week, the White House asked OpenAI and Anthropic to deliver new models to the British AI security institute only after US testing. Washington controls the models’ exit door and leaves the rules about what happens inside them to those who build them.

With Beijing, the risk agenda seemed to come down to a bilateral channel. Treasury Secretary Scott Bessent proposed an incident notification mechanism inspired by the hotline created between Washington and Moscow after the Cuban Missile Crisis. The summit with Xi ended without an agreement on AI. The New York Times reported only the opening of new channels of communication. Five days later, Trump said he does not want to work with China on AI risks because cooperation could undermine America’s lead. ‘Whoever wins superintelligence wins. You’re gonna have a winner and a loser, and you’re probably not gonna have a second place,’ he said.

Xi plays in two registers. At the UN, his ambassador condemned the technological ‘cliques’ that force other countries to pick sides and defended open models. At the White House, Xi accepted the framing of two leading nations with shared responsibility and, according to the Chinese readout, proposed jointly preventing the misuse of AI. Beijing reads the safety agenda of the US labs as an instrument to contain Chinese companies. Minister of State Security Chen Yixin included among the risks of AI the imbalances caused by Western technology monopolies. Researcher Michelle Nie, of the Center for a New American Security, notes that Beijing sees the calls for a pause as a way to inflate the value of US companies before they go public. Anthropic’s prospectus lends substance to that suspicion. The Chinese labs, for their part, were left out of the picture. On the Chinese side, information about flaws flows first to the state. Z.ai passes on the vulnerabilities its systems find to a national database run by the Ministry of State Security.

Who is left behind

The governance vacuum also affects the governments that legislated first. A New York Times survey of more than 20 lawmakers, technologists and experts describes governments outpaced by the technology and companies left, to a large extent, to police themselves. The European AI law exempts tests carried out before a model is launched in Europe. That is why the Hugging Face intrusion fell into a grey area of enforcement. In the US Congress, the bill that would require safety testing of models is stalled along with dozens of other proposals.

Two days before the Security Council session, the UN Independent International Scientific Panel on AI concluded that a failure of agents can cross national borders and that AI safety may be becoming a matter of collective security. That same week, 26 countries and the European Commission, led by Norway, signed the declaration A Call for Control of Frontier AI Models. The text calls for mandatory testing before launch, independent evaluation and shared reporting of serious incidents. It also requires that oversight advance without widening the gap between countries in access to the benefits of AI. It is the only proposal in circulation that makes room for those outside the two poles. The signatories include South Africa, Kenya and Sierra Leone, but not Brazil.

The place of the Global South

At the opening of the General Assembly, Brazilian President Luiz Inácio Lula da Silva accused a handful of techno-oligarchs of running a race without ethical brakes in defiance of the UN. The diagnosis is correct, and the country, like the Global South more broadly, has instruments at hand. The first would be to join the Norwegian declaration, which remains open. Some developing countries are founders of WAICO, the AI cooperation organisation led by Beijing. South Africa, Kenya and Kazakhstan already take part in both initiatives.

In the forums created by the General Assembly in 2025, the Global South can demand that any alert channel between powers or between companies have a multilateral counterpart, with a common registry and mandatory notice to every affected state. The AI bill under debate in the Brazilian Congress, which the New York Times highlights in contrast with the wait-and-see approach of India, Japan, Australia, Canada and Kenya, is the natural vehicle for this obligation. The White House pact shows what happens when such an obligation does not exist. The rule should apply equally to American and Chinese suppliers.

In control of the loss of control

While we marvel at the latest antics of AI agents and speculate about whether the systems may be preparing to destroy us, the ‘owners’ of AI are speeding up the drafting of the rules while rhetorically calling for brakes on development. In the same week as the lunch, Anthropic released its second model since Amodei’s call for a slowdown, and OpenAI unveiled always-on autonomous agents. They are creating a de facto and de jure situation for the control of their own loss of control. As if they were vestal virgins of a dystopian Rome, they believe that governments and the peoples of the world should admire them and create exceptions of every kind so that they can protect us from the risks they themselves created and that, for now, only their employees can mitigate. On Tuesday, the President of the United States took the vow alongside them.

The narrative they are building guarantees them an advantage because whoever invented the poison will offer us the cure. Selling a sense of global panic helps when it is time to present the tranquillisers. And, by setting the technology straight, the labs will be credited with protecting humanity. As happened with internet governance, this recognition of authority may lead countries to take the reins alongside a captured technical community.

Trump and Xi will meet again in November, at the APEC summit in Shenzhen. By then, Trump’s adviser will have delivered the new legal definition of the technology, the labs’ entity will have advanced on its timeline, and the Norwegian declaration will still be waiting for new signatures. If the arrangement consolidates in its current form, AI governance will be in the hands of those with the greatest interest in dictating its rules. The rest of humanity will keep watching from the sidelines.

About the author

James Görgen has been a Public Policy and Government Management Specialist since 2008. He holds a Master’s degree in Communication and Information from UFRGS. He is currently an advisor at Brazil’s Ministry of Development, Industry, Trade, and Services and a member of the Brazilian Internet Steering Committee.

*This article reflects the personal reflections of the author and should not be construed as the official position of the Brazilian Ministry of Development, Industry, Trade, and Services of Brazil.