Statement 37: Establish monitoring framework
Agencies should
Criterion 124: Define reporting requirements.
This includes:
- establishing a plan for providing different stakeholders with reports
- for each group of stakeholders (persona), define what needs to be reported, why, when, and how.
Criterion 125: Define alerting requirements.
This includes:
- defining what information needs alerting
- defining what information is critical to be alerted in real-time
- defining severity levels, such as major, minor, warning
- defining thresholds, out-of-pattern behaviour, and other triggers for each alert level
- defining who needs to be alerted and the method of alert such as SMS or e-mail.
Criterion 126: Implement monitoring tools.
This includes:
- monitoring the information needed to satisfy alerting and reporting requirements
- automating monitoring, alerting, and reporting
- implementing management information and dashboards
- implementing role-based access to protect sensitive information and meet security requirements
- implementing real-time alerting requirements.
Criterion 127: Implement feedback loop to ensure that insights from monitoring are fed back into the development and improvement of the AI system.
This includes:
- a decision matrix outlining guidance on what components in the AI system would need an update or refresh, such as pre or post processing components, AI model, or a RAG knowledge base in a GenAI system
- a framework to provide and track recommended actions from the insights
- a guideline for identifying actions to address insights, with considerations to costs, delays, AI trust, and effectiveness.