Statement 36: Implement rollout and safe rollback mechanisms
Agencies should
Criterion 120: Define a comprehensive rollout and rollback strategy.
This should safeguard data and limit data corruption.
Criterion 121: Implement load balancing and traffic shifting methods for system rollout.
This includes:
- using load balancers to distribute traffic dynamically between old and new deployments during updates
- creating traffic shifting policies to safeguard against overwhelming newly deployed AI systems with high inference demands.
Criterion 122: Conduct regular testing, health checks, readiness, and startup probes to verify stability before routing traffic for all deployed AI services.
Consider using probes to continuously monitor during deployment, to detect issues early and rollback upon failure.
Criterion 123: Implement rollback mechanisms to revert to the last stable version in case of failure.
This includes:
- implementing automated rollback mechanisms to revert to the last stable version in case of pre-defined critical failure for AI deployments
- failures that do not satisfy the trigger for automated rollback require human intervention to analyse and decide the next steps.