AI Audit Accord Needs Real Teeth, Warns Research Head, or Auditors Will Only See What Companies Show Them

On Sept. 29, Nvidia, OpenAI, Anthropic, Google, Meta and xAI signed an accord with the White House on their most advanced AI models, which President Donald Trump called morally binding. Each firm agreed to work with an independent outside auditor and let independent board members review the findings.

Samuel Vance Law of the Decentralization Research Center welcomes the idea but points out that the deal does not say who picks the auditors, what they can examine or whether results become public. There is also no penalty for ignoring it, so each company effectively chooses its own checker and decides what that checker sees.

Lessons From Enron and Weak Voluntary Pledges

Vance-Law recalls that Enron paid Arthur Andersen $25 million for auditing and $27 million for other work in 2000, and both firms collapsed within two years. Congress answered with Sarbanes-Oxley, which removed auditor selection from management and created a public board to inspect auditors. He says the 2023 voluntary AI commitments fared poorly too. Researchers at Stanford, Harvard and Brown scored 16 companies at an average of 53%, and eight scored zero on helping outsiders report problems. METR said its pre-release testing of OpenAI’s o3 lasted three weeks, with limited access and no view of the model’s reasoning.

Double-Blind AI Testing Could Fix the Access Problem

The old dilemma was that testers either revealed their questions or the company handed over a costly model. In August, Google DeepMind and Singapore’s AI Safety Institute, with three other groups, ran a double-blind test on sealed hardware, so testers never saw the model and Google never saw the questions. The risk shifts to the hardware, which researchers once breached with under $1,000 of equipment, though only with physical access.

What Vance-Law Wants Changed

Based on a report that mapped nearly 30 alternatives after 36 interviews, he argues auditors should choose their own tests, companies should name their auditors and describe audit scope, and board committees should answer to someone outside the firm. He says Congress should fill the missing inspector role.