A common funder is not a conspiracy. It is a structural prior.
The AI evaluation landscape has three layers — government safety institutes, Open-Philanthropy-funded academic and nonprofit orgs, and independent civil-society groups. They share findings, methods, and incentives only partially. Knowing which layer a finding came from is half of reading it.
The work of evaluating AI is concentrated in a small, well-named set of organizations. Most of the loudest capability and safety claims come from labs themselves (RSPs, Preparedness frameworks), from government safety institutes (UK AISI, US AISI) that adopted the frontier-AI / x-risk frame in 2023, or from a network of academic and nonprofit evaluators (METR, Apollo, Redwood, Epoch AI, HELM) that share Open Philanthropy as a major funder.
The labor and sociotechnical lens — DAIR, AI Now, AlgorithmWatch, the Algorithmic Justice League — is held mostly by independent civil-society organizations operating at single-digit-million budgets. They produce most of the concrete-harms reporting we covered in Act IV (Sama, Lavender, Robodebt, Worldcoin, ImmigrationOS) but get a fraction of the policy attention given to the safety / capability layer. The asymmetry is structural and worth carrying forward whenever “the experts” are invoked.
Who measures what — and who pays them.
2D scatter: independence (X) × scope (Y). Click any node to see what they track and what they have found. The Y-axis is what they measure; the X-axis is who funds them.
Capability is climbing on a clean exponential.
METR's task-completion-time-horizon doubles every ~7 months. UK AISI's self-replication evals jumped 5%→60% in two years. The straight-line extrapolation is a hypothesis, but the line itself is data.
Five non-obvious findings worth carrying forward.
Each of these has shifted the field's understanding in a way the headlines didn't capture.
The same model name is not the same model.
API-served LLMs change continuously. Most published evals are point-in-time. Treat any 'the model does X' claim with a date stamp.
The GPT-4 prime-identification finding above is the canonical example of a broader pattern: models behind a stable API name are continually retrained, fine-tuned, and patched. There is no public commitment from any frontier lab to versioning that survives policy timelines.
David Rozado's political-bias work (PLOS ONE, 2024) administered 11 political-orientation tests, 10 trials each, to 24 LLMs (2,640 runs). 23 of 24 leaned left; >80% of policy recommendations were left-of-center. Useful as a counterweight to “AI is neutral” framing — though Rozado is at the conservative-leaning Centre for Policy Studies, so methodologically check both ways.
The pragmatic upshot for any AI policy debate: date-stamp every claim about model behaviour, treat capability and bias measurements as snapshots, and read every tracker through its funder.