Two different kinds of assessments
Our proposal for independent verification organizations (IVOs) treats audits and evaluations as two distinct, complementary kinds of assessment that serve different purposes:
- Audits - An assessment of whether the developer has complied with the law, including any minimum standards established by the Governor’s Office of Emergency Services (Cal OES). The main objective is to deter companies from violating the law. We propose that audits be published annually, but that embedded IVOs can report on violations at any time.
- Evaluations - An assessment of catastrophic and security risks from the developer’s models, including recommendations for corrective action. This ensures that an independent party checks whether such risks exist, even when a developer is following every rule. For instance, evaluations can catch risks that outdated minimum standards no longer address or that move too fast for any minimum standard to cover. We propose that evaluations be published quarterly, but that embedded IVOs can report on imminent risks at any time.
Because audits and evaluations call for different expertise and different safeguards for independence, the law should allow different organizations to perform them.
| Audit | Evaluation |
|---|---|
| Measured against an objective standard: the law and Cal OES minimum standards. | No fixed standard; relies on expert technical judgment. |
| Usually performed by an audit firm. | Requires an evaluator with technical expertise on AI. |
| Checks assertions made by the developer’s management. | Generates its own evidence and investigates assertions that management may not have made. |