Statement 25: Implement continuous improvement frameworks
Agencies must
Criterion 89: Establish interface tools and feedback channels for machines and humans.
This also involves providing appropriate human-machine interface tools for human interrogation and oversight.
Criterion 90: Perform model version control.
AI model versioning and tracking is key to comparing performance over time, identifying factors affecting performance and updating the model when needed.
AI model tracking and versioning can involve:
- dataset tracking and versioning
- model tracking and versioning – each trained model can have the following details:
- algorithm, learning type and hyperparameter settings
- compile-time parameters
- the tools or version of tools used to compile the model.
Set-up rollback options to historical models to help develop safety nets in the AI system, reduce risk of deploying new models, and provide deployment flexibility.