11. Accountability
11.1 Ensuring accountability during the life cycle of the AI system
Agencies should consider putting mechanisms in place during the life cycle of the AI system to ensure that the agency itself, or the relevant decision-maker, remains responsible and accountable for a government decision which involves the use of AI. Such mechanisms should clearly define how ultimate responsibility for the decision is retained, even when AI is used to analyse data or generate recommended outcomes.
Accountability should be considered at all stages of the AI system lifecycle. Some of the relevant considerations for different stages are outlined below.
During the design and development phase
- How the AI will be constructed in a way that is consistent with the scope of a decision-maker’s discretion and any legislative framework which confers authority on a decision-maker
- How the AI will be designed in a way that ensures that the decision-maker takes into account any matters which it is required to consider as part of decision-making
- Whether the decision-maker will have the ability to override or disregard decisions made by AI (for example, where its outputs are based on biased data).
During the deployment and operation phase
- What information the decision-maker needs to have oversight of the AI (for example, information on the capacities and limitations of the AI, and the process that the AI will use to reach a conclusion)
- What processes are in place to ensure that, where appropriate, final discretion or judgement lies with the decision-maker (for example, the decision-maker analyses the information provided by the AI before making a decision)
- What records will be kept of the decision-maker’s reasoning at any decision points which require discretion and judgment. This contributes to the contestability of decisions in accordance with section 9.
Where the scope of the use case changes or is developed during the life of the AI system
- What assessment, review and acceptance-testing processes are to be applied to the changes to the AI system to ensure the above.