Statement 35: Deploy to a production environment
Agencies must
Criterion 117: Apply strategies for phased roll-out.
Consider splitting traffic between the current and new version being rolled out, or rolling out to a subset of users to gradually introduce changes and detect issues before full deployment.
Criterion 118: Apply readiness verification, assurance checks, and change management practices for the AI system.
This typically involves:
- the readiness verification, which includes all tests and covers the entire system – code, model, data, and related components
- consent for data governance, data use, and auditing frameworks
- ensuring all production deployments follow change management protocols, including impact assessment, notifying stakeholders, updating training, assurance, approvals, testing, and documentation
- including the rationale for deploying or updating AI systems in the change records to ensure accountability and transparency
- understanding the implications of AI model auto-updates in production, including options to disable
- understanding the implications of AI system online and dynamic learning in production, including options to disable.
Agencies should
Criterion 119: Apply strategies for limiting service interruptions.
This typically involves:
- implementing strategies to avoid service interruptions and reduce risk during updates where zero downtime is required
- configuring instance draining to ensure active requests are not interrupted while allowing completion of long-running AI inference tasks
- include cost tracking on deployment workflows for additional resources used during deployment
- include real-time monitoring and alerting to detect and respond to issues during deployment processes and transitions.