Statement 28: Test for safety, robustness, and reliability
Agencies must
Criterion 101: Test the computational performance of the system.
This includes:
- testing for response times, latency, and resource usage under various loads
- network and hardware load testing.
Criterion 102: Test safety measures through negative testing methods, failure testing, and fault injection.
This includes:
- testing for incorrect or harmful inputs.
Criterion 103: Test reliability of the AI output, through stress testing over an extended period, simulating edge cases, and operating under extreme conditions.
Agencies should
Criterion 104: Undertake adversarial testing (red team testing), attempting to break security and privacy measures to identify weaknesses.
AI-specific attacks can be executed before, during, and after training.
Examples of attacks that can be made before and during training includes:
- dataset poisoning
- algorithm poisoning
- model poisoning
- backdoor attacks.
Examples of attacks that can be made after training includes:
- input attack and evasion
- reverse engineering the model and data.