Executive summary
The state of California should have a way to “pace the frontier” in order to avoid uncontrolled recursive self-improvement, where AI automates the development of AI R&D to such an extent that progress in AI development becomes far too rapid to manage. For this purpose, we recommend that the Governor’s Office of Emergency Services (Cal OES) be given a mandate to (1) measure AI progress, (2) determine a maximum safe rate of progress, and (3) limit AI company computation for certain tasks to keep progress within that safe rate.
Measurement of AI progress
Cal OES will have to measure the rate of AI progress in order to understand its current and historical speed. Cal OES could rely significantly on private sector measures, such as the Epoch Capabilities Index, but it should also have the authority to require additional information from frontier AI developers. This information could include:
- Measures of AI R&D speedup achieved by the use of AI.
- Fractions of AI R&D that are currently automated to varying degrees.
- Results of internal capabilities evaluations for frontier models that have not yet been released and thus do not have scores on public benchmarks.
Cal OES should be as public as possible with its measurements to ensure that they can be publicly interrogated by independent scientists, but may have to keep some information private to protect company trade secrets or guard against companies “gaming” its metrics. Cal OES should consult with relevant independent verification organizations while developing these metrics.
Assessment of a maximum safe rate of progress
Even once the rate of AI progress and recursive self-improvement across the industry is better understood, it is necessary to ask: how fast is too fast? This is a challenging question to answer and it will likely change quickly as both new risks and new safety measures emerge. Cal OES should be authorized to consider the latest science of AI safety and estimate the highest rate of AI progress that would not create unreasonable catastrophic risks to the public. The determination would include a cost-benefit analysis and would include factors such as the possibility that pacing could reduce the lead of California companies in comparison to foreign companies that may not comply with California law.
Compute limitations to keep progress at a safe rate
Once Cal OES has a way to measure the rate of AI progress and has determined (at least preliminarily) what rate would be too fast to be safe, it needs to have a way to apply brakes to the industry to prevent the overall rate of progress from going too fast. This could be analogized to the way that the Federal Reserve sets interest rates in order to control hyperinflation.
The best idea we know of for doing this is compute-based limitations. For example, Cal OES could limit the fraction of frontier AI computational power that AI companies devote to training new models (as opposed to running existing models) or to running models it is using to perform AI research and development. This would have the effect of controlling the rate of AI progress, and Cal OES should set these limits to avoid the rate eclipsing its current safe limit. The exact specifics of the technical limitations should likely not be spelled out in statute to avoid them getting out of date too quickly, but rather through a regulatory process. Compliance with compute limitations would be verified by IVOs.
This provision should not be used to target any one particular company; if one or multiple specific companies are creating serious risks, this should be dealt with by giving Cal OES the power to issue orders to cease that activity, as detailed in our recommendation on orders. This pacing provision should apply to all frontier AI developers at once.