Guidance for the artificial intelligence impact assessment tool

6. Reliability and safety

6.1 Data suitability

The data used to operate, train and validate your AI system has a significant impact on its performance, fairness and safety. In your answer, explain why the chosen data is suitable for your use case. Some relevant considerations are outlined below.

When choosing between datasets, consider whether the data can be separated by marginalised groups, particularly by Indigenous status identifiers. If the data is Indigenous data, see section 6.2 below, you should refer to the Framework for Governance of Indigenous Data.

Agencies should also refer to the Australian Public Service (APS) Data Ethics Framework for guidance on managing and using data and analytics ethically in government, including where AI is used in analytics. The framework is underpinned by 3 key principles: trust, respect and integrity. It provides advice on implementation across different major use cases and agency operations and encourages agencies to assess potential risks and benefits, consider fairness and inclusivity, and engage with stakeholders where appropriate. Visit the Department of Finance website to access the APS Data Ethics Framework.

Data quality should be assessed prior to use in AI systems. Agencies should select applicable metrics to determine a data set's quality and identify any remediation required before using it for training or validation in AI systems. Relevant metrics to consider include diversity, relevance, accuracy, completeness, timeliness, validity and lack of duplication. One method to ensure good quality data is to set minimum thresholds appropriate to specific use cases, such as through acceptance criteria discussed below at section 6.4. An example of a specific framework for determining data quality in statistical uses is the ABS Data Quality Framework.

Where third party material or data is being used to operate, train or validate an AI system, it is important to protect the rights of intellectual property holders. If the AI may use, modify or otherwise handle material in which intellectual property exists, agencies should confirm that both the following are true:

• the AI provider holds the necessary intellectual property rights in the AI output material
• the agency holds the necessary intellectual property rights in the input material.

The AI may otherwise infringe third party intellectual property rights.

Agencies should also confirm that the AI system has safeguards in place to prevent the unauthorised use or disclosure of confidential information.

Where data used to operate, train and validate the AI system includes personal information, agencies should confirm that collection, use and disclosure is in accordance with the Australian Privacy Principles (APPs) under the Privacy Act 1988 (Cth) (see section 7 of this guidance).

The relevance of the data used in training the AI model may influence the output and may not be relevant to the use case and Australian context. Consider whether the model is likely to make accurate or reliable predictions concerning matters relating to Australian subject matter if it has been trained on, for example, US centric data.

You should also consider data provenance, lineage and volume – as outlined below:

Data provenance

Involves keeping records of the data collected, processed and stored by the AI system and creating an audit trail to assign custody and trace accountability for issues. It provides assurance of the chain of custody and its reliability, insofar as origins of the data are documented.

Data lineage

Involves documenting data origins and flows to enable stakeholders to better understand how datasets are constructed and processed. This fosters transparency and trust in AI systems.

Data volume

Consider the volume of data you need to support the operation, training and validation of your AI system.

6.2 Indigenous data

Describe how any components of your AI system have used or will use Indigenous data, or where any outputs relate to First Nations individuals, communities or groups.

All Australian Public Service (APS) agencies are required to implement the Framework for Governance of Indigenous Data. This framework adopts the definition of 'Indigenous data' as provided by Maiam nayri Wingara Indigenous Data Sovereignty Collective:

Information or knowledge, in any format or medium, which is about and may affect Indigenous peoples both collectively and individually.

If the data used to operate, train or validate your AI system, or any outputs from your AI system, meet this definition of Indigenous data, refer to the Framework for Governance of Indigenous Data for guidance on applying the framework.

The framework is based on the principles of:

• respect for cultural heritage
• informed consent
• privacy, including collective or group privacy
• trust.

The Framework for Governance of Indigenous Data is also informed by 2 complementary data governance frameworks:

• FAIR Guiding Principles (Findable, Accessible, Interoperable, Reusable) – providing technical standards for scientific data management and stewardship
• CARE Principles (Collective Benefit, Authority to Control, Responsibility, Ethics) – which focuses on Indigenous data governance, reflecting the crucial role of data in self-determination.

Relevant practices to consider in this context include:

• Checking if datasets used to train the AI included diverse and representative samples of cultural expression, artifacts, languages and practices. This supports the AI system being able to recognise and appropriately respond to a greater range of cultural contexts in a less biased manner.
• Describing any mechanisms in place for engaging with Indigenous individuals, communities or group representatives and collecting and incorporating their feedback on the AI system's performance, especially regarding cultural aspects.
• Describing processes to review documentation and protocols that ensure the project has incorporated the GID principles. Look for evidence of meaningful engagement with and input from suitably qualified and experienced Indigenous individuals, communities and groups. Assess if the system includes features or options that allow Indigenous stakeholders to control how their data is used and represented and describe how benefits of the project to First Nations Peoples, to which the data relate, have been considered.

Also consider the use of Indigenous data in the context of the United Nations Declaration on the Rights of Indigenous Peoples and apply the concept of 'free, prior and informed consent' in relation to the use of Indigenous data in AI systems.

6.3 Suitability of procured AI model

If you are procuring an AI model or system from a third-party provider, your procurement process should consider if the provider has appropriate data management including data quality and data provenance in relation to the model. This will help you to identify whether the AI model is fit for the context and purpose of your AI use case.

This may include:

• governance
• data sourcing
• privacy
• security
• intellectual property
• cybersecurity practices.

There are many other considerations you should take into account when selecting a procured AI model and contracting with a supplier. The following considerations may be relevant to your use case:

• Determine if data will be hosted overseas and if it could be subject to foreign laws. Consider the potential for foreign ownership, control, or influence (FOCI) and refer to the Department of Home Affairs FOCI Risk Assessment Guidance.
• Determine if processes and practices are in place to address risks along the supplier's supply chain, such as risks relating to FOCI, security, transparency and business practices). Agencies should refer to Australian Signals Directorate guidance on cyber supply chain risk management.
• Assess whether the AI model meets the functional requirements for your use case.
• Determine how the model was evaluated, including the test data and benchmarks used.
• Determine how versioning for the AI model is handled.
• Consider the support the supplier provides for users and procurers.
• Review provisions regarding potential liability issues and clarify accountability between your agency and the provider if the product fails.
• Establish security precautions, such as handover or destruction of agency data upon termination or expiry of the procurement contract, and identify any residual risks and mitigation measures.
• Confirm what controls the agency has if the AI system malfunctions, produces harmful outputs or behaves in an unintended way.
• Review any guarantees that data handling and management across the entire lifecycle of the data meet internal agency and legislative requirements.
• Review any warranties the supplier will provide, such as suitability of the AI system for the intended use, absence of defects, and development with reasonable care and skill.
• Ensure the supplier has a contractual obligation to comply with relevant legislation and frameworks, including privacy, discrimination, AI Ethics Principles, the AI policy and the Protective Security Policy Framework (PSPF).
• Clarify any supplier responsibilities for training, monitoring and validation of the AI system.
• Clarify ownership of intellectual property rights in relation to the AI model, inputs, outputs and other materials such as user manuals or technical documentation.
• Review measures taken to prevent or reduce hallucinations, unwanted bias and model drift. For example, evaluation of training data for harm or bias and adjustments made to compensate.
• Assess whether the level of human oversight, transparency, explainability and interpretability of the model is sufficient for your use case.
• Specify the kinds of records the supplier will provide to the agency, such as records of how agency data is used by the AI system.
• Determine the computing and storage capacity requirements for operating the model on premises.
• Assess the capability needed to maintain the AI model and whether this can be done in-house or require external sourcing.
• If using a platform as a service (PaaS) to run and support your AI system or AI model, consider risks associated with outsourcing.
• Evaluate whether the AI system could be designed or influenced to promote certain products or services, and how such behaviour could be detected and addressed. For example, if the supplier accepts advertising or sponsorship to give prominence to products or services.

Consider also how your agency will support transparency across the AI supply chain, for example, by notifying the developer of issues encountered in using the model or system. Refer to the DTA's AI procurement resources including the:

• Guidance on AI procurement in government
• AI contract template
• Digital Sourcing ClauseBank AI model clauses.

6.4 Testing

Testing is a key element for assuring the responsible and safe use of AI models – for both models developed in-house and externally procured – and in turn, of AI systems. Rigorous testing helps validate that the system performs as intended across diverse scenarios. Thorough and effective testing helps identify problems before deployment.

Testing AI systems against test datasets can reveal biases or possible unintended consequences or issues before real-world deployment. Testing on data that is limited or skewed can fail to reveal shortcomings.

Consider establishing clear and measurable acceptance criteria for the AI system that, if met, would be expected to control harms that are relevant in the context of your AI use case. Acceptance criteria should be specific, objective and verifiable. They are meant to specify the conditions under which a potential harm is adequately controlled.

Consider developing a test plan for the acceptance criteria to outline the proposed testing methods, tools and metrics. Documenting results through a test report will assist with demonstrating accountability and transparency. A test report could include the following:

• a summary of the testing objectives, methods and metrics used
• results for each test case
• an analysis of the root causes of any identified issues or failures
• recommendations for remediation or improvement, and whether the improvements should be done before deployment or as a future release.

In your explanation, outline any areas of concern in results from testing. If the AI system has not yet undergone testing, outline elements to be considered in testing plans.

Model accuracy

As an example. model accuracy is a key metric for evaluating the performance of an AI system. Accuracy should be considered in the specific context of the AI use case, as the consequences of errors or inaccuracies can vary significantly depending on the domain and application. This can include:

• unfairness – for example, where a decision has been made based on inaccurate data
• breach of individual rights – for example, where information produced by AI is defamatory
• non-compliance with legislation – for example, presenting false or misleading information in breach of Australian Consumer Law, or acting in a discriminatory manner in breach of anti-discrimination laws.

Some of the factors that can influence AI model output accuracy and reliability include:

• choice of AI model or model architecture
• quality, accuracy and representativeness of training data
• presence of bias in the training data or AI model
• robustness to noise, outliers and edge cases
• ability of the AI model to generalise to new data
• potential for errors or 'hallucinations' in outputs
• environmental factors (such as lighting conditions for computer vision systems)
• adversarial attacks (such as malicious actors manipulating input data to affect outputs)
• stability and consistency of performance over time
• whether AI model allows for sponsorship or advertising to give prominence to certain outputs.

Ways to assess and validate the accuracy of your model for your AI use case include:

• quantitative metrics
• qualitative analysis – such as manual review of output, error analysis, and user feedback
• domain-specific benchmarks or performance standards
• comparison to human performance or alternative models.

It is important to set accuracy targets that are appropriate for the risk and context of the use case. For high stakes decisions, you should aim for a very high level of accuracy and have clear processes for handling uncertain or borderline cases.

6.5 Pilot

Conducting a pilot study is a valuable way to assess the real-world performance and impact of your AI use before full deployment. A well-designed pilot can surface issues related to reliability, safety, fairness and usability that may not be apparent in a controlled development environment.

If you are planning a pilot, your explanation should provide a brief overview of the pilot's:

• scope and duration
• objectives and key results (OKRs)
• key performance indicators (KPIs)
• participant selection and consent process
• risk mitigation strategies.

If you have already completed a pilot, reflect on the key findings and lessons learned, including by:

• assessing how the pilot outcomes compared to your expectations.
• identifying any issues or surprises that emerged during the pilot.
• documenting how you adapted your AI use case based on the pilot results.

If you are not planning to conduct a pilot, explain why not. Consider whether the scale, risk or novelty of your use case warrants a pilot phase. Discuss alternative approaches you are taking to validate the performance of your AI use case and gather user feedback prior to full deployment.

6.6 Monitoring

Monitoring is key to maintaining the reliability and safety of AI systems over time. It enables active rather than passive oversight and governance, and ensures the agency has ongoing accountability for the AI-assisted performance and decision-making processes.

Your monitoring plan should be tailored to the specific risks and requirements of your use case. In your explanation, describe your approach to monitoring any measurable acceptance criteria (as discussed above at section 6.4) and other relevant metrics such as performance metrics or anomaly detection. In your plan, include your proposed monitoring intervals for your use case. The AI policy requires agencies to establish a clear process to address AI incidents aligned to their ICT management approach. Incident remediation must be overseen by an appropriate governance body or senior executive and should be undertaken in line with any other legal obligations.

Periodically evaluate your monitoring and evaluation mechanisms to ensure they remain effective and aligned with evolving conditions throughout the lifecycle of your AI use case. Examples of events that could influence your monitoring plan are system upgrades, error reports, changes in input data, performance deviation or feedback from stakeholders.

Monitoring can help identify issues that can impact the safety and reliability of your AI system, such as:

• concept drift – changes in the relationship between input data and the feature being predicted
• data drift – changes in input data patterns compared to the data used to train the model.

Vendors offer monitoring tools that may be worth considering for your use case. For more information on continuous monitoring, refer to the NAIC's Implementing Australia's AI Ethics Principles report.

6.7 Preparedness to intervene or disengage

Relevant stakeholders, including those who operate, use or interact with the AI system, those who monitor AI system performance, and affected stakeholders identified at section 2.4, should have the ability to raise concerns about insights or decisions assisted by the AI system.

Agencies must develop clear pathways for staff or other relevant stakeholders to report AI safety concerns, including AI incidents. Agencies should also document and take appropriate steps in relation to any interventions that occur to ensure consistency and fairness.

In addition, agencies should be prepared to quickly and safely disengage an AI system when an unresolvable issue is identified. This could include a data breach, unauthorised access or system compromise. Consider such scenarios in business continuity, data breach and security response plans.

Techniques to avoid overreliance on AI system outputs

Agencies should consider the following techniques to avoid overreliance on AI system outputs.

Three techniques to consider at the system design stage:

• Build in transparency about system limitations, by incorporating prompts to remind users to critically analyse outputs. These could include explanations of outputs, hallucination reminders, reference source checking and accuracy scores.
• Build in 2-way feedback pathways by prompting users to assess the quality of the AI system's outputs and provide feedback. Similarly, provide feedback to users on their interactions with the systems, such as feedback on ineffective prompts or alerts when the user has accepted a risky decision.
• Build in steps that require human decision-making, for example by designing the AI system to provide options to choose from rather accept a single outcome, prompting users to engage with and evaluate AI outputs.

At the evaluation stage, focus on validating whether the system supports human judgement as intended. Engage directly with users to understand their experience, encourage them to assess outputs critically and suggest improvements. Review user behaviour, feedback loops and decision-making patterns and prompts to confirm that safeguards against overreliance are effective. Use these insights to refine system design, guidance and training materials.

6.8 Training of AI system operators

AI system operators play a crucial role in ensuring the responsible and effective use of AI. They must have the necessary skills, knowledge and judgment to understand the system's capabilities and limitations, how to appropriately use the system, interpret its outputs and make informed decisions based on those outputs.

In your answer, describe the process for ensuring AI system operators are adequately trained and skilled. This may include:

Initial training

Consider what training operators receive before being allowed to use the AI system. Does this training cover technical aspects of the system, as well as ethical and legal considerations?

As a baseline, you may expect that operators:

• understand the limitations of the AI system
• are able to monitor the AI system, so that anomalies, errors and unexpected performance can be detected and addressed
• are aware of the possible tendency of relying, or over-relying, on AI outputs
• are able to correctly interpret AI outputs, taking into account the particular characteristics of the system
• are able to decide when to disregard, override or reverse the AI outputs.

Ongoing training

This includes processes for continuous learning and skill development, and for keeping officers up to date with changes or updates to the AI system.

Evaluation

This can include skills and knowledge assessment, certification or qualification requirements for operators.

Support

Ensure resources and support are available to operators if they have questions or encounter issue. Consider whether this needs to be tailored to the specific needs and risks of your AI system or proposed use case or whether general AI training requirements are sufficient.