Search

2. Purpose and expected benefits

2.1    Problem definition

Describe the problem that you are trying to solve. 

For example, the problem might be that your agency receives a high volume of public submissions, and that this volume makes it difficult to engage with the detail of issues raised in submissions in a timely manner.

Do not describe how you plan to fix the problem or how AI will be used. 

Though ‘problem’ implies a negative framing, the problem may be that your agency is not able to take full advantage of an opportunity to do things in a better or more efficient way. 

2.2    AI use case purpose

Clearly and concisely describe the purpose of your use of AI, focusing on how it will address the problem you described at 2.1. 

Your answer may read as a positive restatement of the problem and how it will be addressed.

For example, the purpose may be to enable you to process public submissions more efficiently and effectively and engage with the issues that they raise in more depth. 

2.3    Non‑AI alternatives

Briefly outline non‑AI alternatives that could address the problem you described at 2.1. 

Non‑AI alternatives may have advantages over solutions involving AI. For example, they may be cheaper, safer or more reliable. 

Considering these alternatives will help clarify the benefits and drawbacks of using AI and help your agency make a more informed decision about whether to proceed with an AI‑based solution. 

2.4    Identifying stakeholders

Conduct a mapping exercise to identify the individuals or groups who may be affected by the AI use case. Consider holding a workshop or brainstorm with a diverse team to identify the different direct and indirect stakeholders of your AI use case. 

The list below may help generate discussion on the types of stakeholder groups to consider. Please note the stakeholder types below have been provided as a prompt to aid discussion and is not intended as a prescriptive or comprehensive list.

For each type, identify the use case stakeholders and how they might be affected (positively or negatively).

End users

People who will use the AI system and/or interpret its outputs.

Evaluation or decision subjects

People or groups who will be evaluated or monitored by the system (e.g. who the system is making predictions or recommendations about).

Oversight team

The person or team who is managing, operating, overseeing or controlling and monitoring the system during and after deployment.

System owner or deployer 

The executive executives responsible for making decisions on whether to use a system for a particular use.

AI model or AI system engineers

Those involved in AI model or system design, development and maintenance.

Rights holders

Those who hold the rights to materials used by AI (e.g. copyright owners or creators).

Malicious actors

Those who may intentionally misuse the system.

Bystanders

Those in vicinity of system that may be impacted.

Regulators and civil society organisations

Those who regulate, advocate for regulation, or are concerned about compliance.

Communities, or groups

Communities who are likely to be affected by the use of the system.

Associated parties

Third parties impacted by an evaluation or decision and other stakeholders who may have an interest in the use of the system based on their relationship to other stakeholders.

Non-end-user APS staff

APS staff whose roles and workflows will be affected by AI but are not end users of your AI use case.

Intermediaries

A facilitator or agent between 2 parties whose role may evolve with AI integration (e.g. tax agents).

2.5    Expected benefits

This section requires you to explain the expected benefits of the AI use case, considering the stakeholders identified in the previous question.

This analysis should be supported by specific metrics or qualitative analysis. Metrics should be quantifiable measures of positive outcomes that can be measured after the AI is deployed to assess the value of using AI. Any qualitative analysis should consider whether there is an expected positive outcome and whether AI is a good fit to accomplish the relevant task, particularly when compared to the non‑AI alternatives you identified previously. Benefits may include gaining new insights or data. 

Consider consulting the following resources for further advice

• Benefits management policy
• Australian Government Guide to Policy Impact Analysis.

3. Threshold assessment

3.1    Threshold assessment process

To complete the threshold assessment, follow these steps.

3.1.1   Determine likelihood and consequence

For each risk category listed in the assessment, determine the likelihood and consequence of the risk occurring for your AI use case. You should consult the likelihood and consequence descriptors at the Attachment to this guidance.

The risk assessment should reflect the intended scope, function and risk controls of the AI use case.

In conducting your assessment, you should be clear on:

• key factors contributing to the likelihood and consequence of the risk
• how any existing or planned risk controls contribute to the likelihood and consequence of the risk
• any assumptions or uncertainties affecting your risk assessment.

3.1.2    Determine risk severity

Use the risk matrix provided in the framework and at the attachment to this guidance to determine the risk severity for each category.

3.1.3    Provide explanations

In the ‘rationale’ column, provide a clear and concise explanation for each risk rating (aim for no more than 200 words per risk but use additional words if necessary).

You should cover the factors, controls and assumptions outlined above at step 1.

3.2    Assessment contact officer recommendation

Once completed, if the Assessment Contact Officer is satisfied that all risks are low, they may recommend that a full assessment is not required and that the executive sponsor accept the low risks and endorse the use case. If one or more risks are medium or higher, the assessment contact officer must either: 

• complete a full assessment
• amend the scope, function or risk controls to a point where the threshold assessment results in a low risk rating
• decide to not accept the risk and not proceed with the AI use case.

3.3    Executive sponsor review

Once the assessment contact officer has made their recommendation, the executive sponsor must: 

1. review the recommendation 
2. confirm whether they are satisfied by the supporting analysis 
3. agree that a full assessment is or is not necessary for the use case.

When completing the threshold assessment, keep in mind the following:

• Try to be objective and honest in your assessment of risks. Underestimating risks at this stage could lead to inadequate risk management.
• Determining risk ratings can be challenging. Seek guidance from others to assist you (especially subject matter experts and those experienced in safe and responsible AI risk assessments).
• Consider the perspectives of stakeholders, including those identified at section 2.4, in assessing the likelihood and consequence of risks. 
  • Ensure you consider the perspectives of marginalised groups, including First Nations people, especially in relation to the risks relating to discrimination and stereotyping. You may not have the background or life experience to fully appreciate these risks.
• Where there is uncertainty or disagreement about the appropriate risk severity rating, err on the side of caution and choose the higher rating.
• Document key assumptions or evidence used in determining the risk severity ratings, as this will help explain the rationale for your assessment to reviewers.
• Consider the expected benefits of the AI use case before deciding whether to proceed based on significant but mitigable risks.

4. Fairness

4.1   Defining fairness

Fairness is a core principle in the design and use of AI systems, but it is a complex and contextual concept. Australia’s AI Ethics Principles state that AI systems should be inclusive and accessible and should not involve or result in unfair discrimination. However, there are different and sometimes conflicting definitions of fairness, and people may disagree on what is fair. 

For example, there is a distinction between individual fairness (treating individuals similarly) and group fairness (similar outcomes across different demographic groups). Different approaches to fairness involve different trade‑offs and value judgments. The most appropriate fairness approach will depend on the specific context and objectives of your AI use case.

When defining fairness for your AI use case, you should be aware that AI models are typically trained on broad sets of data that may contain bias. Bias can arise in data where it is incomplete, unrepresentative or reflects societal prejudices. AI models may reproduce biases present in the training data, which can lead to misleading or unfair outputs, insights or recommendations.

This may disproportionally impact some groups, such as First Nations people, people with disability, LGBTIQ+ communities and multicultural communities. For example, an AI tool used to screen job applicants might systematically disadvantage people from certain backgrounds if trained on hiring data that reflects past discrimination.

When defining fairness for your AI use case, it is recommended that you:

• consult relevant domain experts, affected parties and stakeholders (such as those you have identified at section 2.4) to help you understand the trade‑offs and value judgements that may be involved
• document your definition of fairness in your response to section 4.1, including how you have balanced competing priorities and why you believe it to be appropriate to your use case
• be transparent about your fairness definition and be open to revisiting it based on stakeholder feedback and real‑world outcomes.

You should also ensure that your definition of fairness complies with anti‑discrimination laws. In Australia, it is unlawful to discriminate on the basis of a number of protected attributes including age, disability, race, sex, intersex status, gender identity and sexual orientation in certain areas of public life, including education and employment. Australia’s federal anti‑discrimination laws are contained in the following legislation:

• Age Discrimination Act 2004
• Disability Discrimination Act 1992
• Racial Discrimination Act 1975
• Sex Discrimination Act 1984.

Resources

• Resources on fairness in AI from the OECD Catalogue of Tools & Metrics for Trustworthy AI 
• Fairness Assessor Metrics Pattern from the CSIRO Data61 Responsible AI Pattern Catalogue

4.2   Measuring fairness 

You may be able to use a combination of quantitative and qualitative approaches to measuring fairness. Quantitative fairness metrics can allow you to compare outcomes across different groups and assess this against fairness criteria. Qualitative assessments, such as stakeholder engagement and expert review, can provide additional context and surface issues that metrics alone might miss.

Quantifying fairness

The specific quantitative metrics you use to measure fairness will depend on the definition of fairness you have adopted for your use case. When selecting fairness metrics, you should:

• choose metrics that align with your fairness definition, recognising the trade‑offs between different fairness criteria and other objectives like accuracy
• confirm whether you have appropriate data to assess those metrics, including sensitive attributes where appropriate (see Australian Privacy Principles 3.3)
• set clear and measurable acceptance criteria (see guidance for 5.4) 
• establish a plan for monitoring these metrics (see 5.6) and processes for remediation, intervention or safely disengaging the AI system if those thresholds are not met.

For examples of commonly used fairness metrics, see the Fairness Assessor Metrics in CSIRO Data61’s Responsible AI Pattern Catalogue.

Qualitatively assessing fairness

Consider some of these qualitative approaches, which may be useful to overcome data limitations and to surface issues that metrics may overlook.

Stakeholder engagement

Consult affected communities, stakeholders and domain experts to understand their perspectives and identify potential issues.

User testing and feedback

Test your AI system with diverse users and solicit their feedback on the fairness and appropriateness of the system’s outputs. Seek out the perspectives of marginalised groups and those groups that may be impacted by the AI system.

Expert review

Engage experts, such as AI ethicists or accessibility and inclusivity specialists, to review the fairness of your system’s outputs and the overall fairness approach and identify potential gaps or unintended consequences.

Resources

• Implementing Australia’s AI Ethics Principles: provides tools and techniques for measuring and minimising bias in AI systems
• List of fairness metrics at Supplementary Table 1 of the research paper A translational perspective towards clinical AI fairness
• Resources on fairness in AI from the OECD Catalogue of Tools & Metrics for Trustworthy AI 
• Fairness Assessor Metrics Pattern from the CSIRO Data61 Responsible AI Pattern Catalogue

6. Privacy protection and security

6.1   Minimise and protect personal information

Data minimisation

Data minimisation is an important consideration when developing and deploying AI systems for several reasons, including privacy and improving quality and model stability. In some cases, more data may be warranted (for example, some large language models) but it is important that you follow good practice in determining the data needed for your use case. 

Privacy requirements for personal information under the Australian Privacy Principles (APPs) are an important consideration in responding to this question. Ensure you have considered your obligations under the APPs, particularly APPs 3, 6 and 11.

For more information, you should consult the APP guidelines, your agency’s internal privacy policy and resources and privacy officer.

Privacy enhancing technologies

Your agency may want or need to use privacy enhancing technologies to assist in de‑identifying personal information under the APPs or as a risk mitigation/trust building approach. Under the Privacy Act 1988 (Cth) and the APPs, where information has been appropriately de‑identified it is no longer personal information and can be used in ways that the Privacy Act would normally restrict. 

The Office of the Australian Information Commissioner’s (OAIC) website provides detailed guidance on De-identification and the Privacy Act that agencies should consider. You may also wish to refer to the De-identification Decision-Making Framework, jointly developed by the OAIC and CSIRO Data61.

6.2   Privacy assessment

The Australian Government Agencies Privacy Code (the Privacy Code) requires Australian Government agencies subject to the Privacy Act 1988 to conduct a privacy impact assessment (PIA) for all ‘high privacy risk projects’. A project may be a high privacy risk if the agency reasonably considers that the project involves new or changed ways of handling personal information that are likely to have a significant impact on the privacy of individuals.

A Privacy Threshold Assessment (PTA) is a preliminary assessment to help you determine your project’s potential privacy impacts and give you a sense of the risk level, including whether it could be a ‘high privacy risk project’ requiring a PIA under the Code. 

This assurance framework does not determine the timing for conducting a PIA or PTA – it may be appropriate that you conduct a PIA or PTA earlier than your assessment of the AI use case under this framework.

If no PIA or PTA has been undertaken, explain why and what consideration there has been of potential privacy impacts.

Privacy assessments should consider if relevant individuals have provided informed consent, where required, to the collection, sharing and use of their personal information in the AI system’s training, operation or as an output for making inferences. Also consider how any consent obtained, including a description of processes used to obtain the consent, has been recorded.

For more information, you should consult the guidance on the Office of the Australian Information Commissioner’s website. You can also consult your agency’s privacy officer and internal privacy policy and resources.

If your AI system has used or will use Indigenous data, you should also consider whether notions of ‘collective’ or ‘group’ privacy of First Nations people are relevant and refer to the guidelines in the Framework for Governance of Indigenous Data (see 5.2).

6.3   Authority to operate

The Protective Security Policy Framework (PSPF) applies to non‑corporate Commonwealth entities subject to the Public Governance, Performance and Accountability Act 2013 (PGPA Act). 

Refer to the relevant sections of the PSPF on safeguarding information and communication technology (ICT) systems to support the secure and continuous delivery of government business. 

Under the PSPF, entities must effectively implement the Australian Government Information Security Manual (ISM) security principles and must only use ICT systems that the determining authority (or their delegate) has authorised to operate based on the acceptance of the residual security risks associated with its operation.

In addition, the Australian Signals Directorate’s Engaging with Artificial Intelligence guidance outlines mitigation considerations for organisations to consider. It is highly recommended that your agency engages with and implements the mitigation considerations in the guidance. 

AI systems that have already been authorised or fall within existing authorisations by your agency’s IT Security Adviser (ITSA) do not have to be re‑authorised. 

It is recommended you engage with your agency’s ITSA early to ensure all PSPF and ISM requirements are fulfilled. 

7. Transparency and explainability

7.1    Consultation

You should consult with a diverse range of internal and external stakeholders at every stage of your AI system’s deployment to help identify potential biases, privacy concerns, and other ethical and legal issues present in your AI use case. This process can also help foster transparency, accountability, and trust with your stakeholders and can help improve their understanding of the technology’s benefits and limitations. Refer to the stakeholders you identified in section 2.4.

If your project has the potential to significantly impact Aboriginal and Torres Strait Islander peoples or communities, it is critical that you meaningfully consult with relevant community representatives.

Consultation resources

APS Framework for Engagement and Participation – sets principles and standards that underpin effective APS engagement with citizens, community and business and includes practical guidance on engagement methods.

Office of Impact Analysis Best Practice Consultation guidance note – provides a detailed explanation of the application of the whole-of-government consultation principles outlined in the Australian Government Guide to Policy Impact Analysis.

AIATSIS Principles for engagement in projects concerning Aboriginal and Torres Strait Islander peoples – provides non-Indigenous policy makers and service designers with the foundational principles for meaningfully engaging with Aboriginal and Torres Strait Islander peoples on projects that impact their communities.

7.2    Public visibility 

Where appropriate, you should make the scope and goals of your AI use case publicly available. You should consider publishing relevant, accessible information about your AI use case in a centralised location on your agency website. This information could include:

• use case purpose
• overview of model and application
• benefits
• risks and mitigations
• training data sources
• compliance with the Policy for the responsible use of AI in government
• contact officer information.

Note: All agencies in scope of the Policy for the responsible use of AI in in government are required to publish an AI transparency statement. More information on this requirement can be found in the policy and associated guidance. You may wish to include information about your use case in your agency’s AI transparency statement. 

Considerations for publishing

In some circumstances it may not be appropriate to publish detailed information about your AI use case. When deciding whether to publish this information you should balance the public benefits of AI transparency with the potential risks as well as compatibility with any legal requirements around publication. 

For example, you may choose to limit the amount of information you publish or not publish any information at all if:

• the AI use case is still in the experimentation phase
• publishing may have negative implications for national security
• publishing may have negative implications for criminal intelligence activities
• publishing may significantly increase the risk of fraud or non-compliance
• publishing may significantly increase the risk of cybersecurity threats
• publishing may jeopardise commercial competitiveness.

7.3    Maintain appropriate documentation and records 

You may also wish to refer to the exemptions under the Freedom of Information Act 1982 in considering whether it is appropriate to publish information about your AI use case.

Agencies should comply with legislation, policies and standards for maintaining reliable and auditable records of decisions, testing, and the information and data assets used in an AI system. This will enable internal and external scrutiny, continuity of knowledge and accountability. This will also support transparency across the AI supply chain – for example, this documentation may be useful to any downstream users of AI models or systems developed by your agency.

Agencies should document AI technologies they are using to perform government functions as well as essential information about AI models, their versions, creators and owners. In addition, artifacts used and produced by AI – such as prompts, inputs and raw outputs – may constitute Commonwealth records under the Archives Act 1983 and may need to be kept for certain periods of time identified in records authorities issued by the National Archives of Australia (NAA).

To identify their legal obligations, business areas implementing AI in agencies may want to consult with their information and records management teams. The NAA can also provide advice on how to manage data and records produced by different AI use cases. 

The NAA Information Management Standard for Australian Government outlines principles and expectations for the creation and management of government business information. Further guidance relating to AI records is available on the NAA website under Information Management for Current, Emerging and Critical Technologies.

AI documentation types

Where suitable, you should consider creating the following forms of documentation for any AI system you build. If you are procuring an AI system from an external provider, it may be appropriate to request these documents as part of your tender process.

System factsheet/model card

A system factsheet (sometimes called a model card) is a short document designed to provide an overview of an AI system to non-technical audiences (such as users, members of the public, procurers, and auditors). These factsheets usually include information about the AI system’s purpose, intended use, limitations, training data, and performance against key metrics. 

Examples of system factsheets include Google Cloud Model Cards and IBM AI factsheets.

Datasheets

Datasheets are documents completed by dataset creators to provide an overview of the data used to train and evaluate an AI system. Datasheets provide key information about the dataset including its contents, data owners, composition, intended uses, sensitivities, provenance, labelling and representativeness.

Examples of datasheets include Google’s AI data cards and Microsoft’s Aether Data Documentation template.

System decision registries

System decision registries record key decisions made during the development and deployment of an AI system. These registries contain information about what decisions were made, when they were made, who made them and why they were made (the decision rationale). 

Examples of decision registries include Atlassian’s DACI decision documentation template and Microsoft’s Design Decision Log.

Documentation in relation to reliability and safety

It is also best practice to maintain documentation on testing, piloting and monitoring and evaluation of your AI system and use case, in line with the practices outlined in section 5.

See Implementing Australia’s AI Ethics Principles for more on AI documentation.

7.4    Disclosing AI interactions and outputs

You should design your use case to inform people (including members of the public, APS staff and decision-makers) that that they are interacting with an AI system or are being exposed to content that has been generated by AI.

When to disclose use of AI

You should ensure that you disclose when a user is directly interacting with an AI system, especially: 

• when AI plays a significant role in critical decision-making processes
• when AI has potential to influence opinions, beliefs or perceptions
• where there is a legal requirement regarding AI disclosure
• where AI is used to generate recommendations for content, products or services.

You should ensure that you disclose when someone is being exposed to AI-generated content and:

• any of the content has not been through a contextually appropriate degree of fact checking and editorial review by a human with the appropriate skills, knowledge or experience in the relevant subject matter
• the content purports to portray real people, places or events or could be misinterpreted that way
• the intended audience for the content would reasonably expect disclosure.

Exercise judgment and consider the level of disclosure that the intended audience would expect, including where AI-generated content has been through rigorous fact-checking and editorial review. Err on the side of greater disclosure – norms around appropriate disclosure will continue to develop as AI-generated content becomes more ubiquitous.

Mechanisms for disclosure of AI interactions:

When designing or procuring an AI system, you should consider the most appropriate mechanism(s) for disclosing AI interactions. Some examples are outlined below:

Verbal or written disclosures

Verbal or written disclosures are statements that are heard by or shown to users to inform that they are interacting with (or will be interacting with) an AI system. 

For example, disclaimers, warnings, specific clauses in privacy policy and/or terms of use, content labels, visible watermarks, by-lines, physical signage, communication campaigns. 

Behavioural disclosures

Behavioural disclosure refers to the use stylistic indicators that help users to identify that they are engaging with AI-generated content. These indicators should generally be used in combination with other forms of disclosure.

For example, using clearly synthetic voices or formal, structured language, robotic avatars.

Technical disclosures

Technical disclosures are machine-readable identifiers for AI‑generated content.

For example, inclusion in metadata, technical watermarks, cryptographic signatures.

Agencies should consider using AI systems that use industry-standard provenance technologies, such as those aligned with the standard developed by the Coalition for Content Provenance (C2PA). 

7.5    Offer appropriate explanations

Explainability refers to accurately and effectively conveying an AI system’s decision process to a stakeholder, even if they don’t fully understand the specifics of how the model works. Explainability facilitates transparency, independent expert scrutiny and access to justice.

You should be able to clearly explain how a government decision or outcome has been made or informed by AI to a range of technical and non-technical audiences. You should also be aware of any requirements in legislation to provide reasons for decisions, both generally and in relation to the particular class of decisions that you are seeking to make using AI.

Explanations may apply globally (how a model broadly works) or locally (why the model has come to a specific decision). You should determine which is more appropriate for your audience. 

Principles for providing effective explanations

Contrastive

Outline why the AI system output one outcome instead of another outcome.

Selective

Focus on the most-relevant factors contributing to the AI system’s decision process.

Consistent with the audience’s understanding

Align with the audience’s level of technical (or non-technical) background. 

Generalisation to similar cases

Generalise to similar cases to help the audience predict what the AI system will do.

You may wish to refer to Interpretable Machine Learning: A Guide for Making Black Box Models Explainable for further advice and examples.

Tools for explaining non-interpretable models

While explanations for interpretable models (i.e. low complexity with clear parameters) are relatively straightforward, in practice most AI systems have low interpretability and require effective post-hoc explanations that strike a balance between accuracy and simplicity. Among other matters, agencies should also consider what are appropriate timeframes for explanations to be provided in the context of their use case. 

Below are some tools or approaches that can assist with developing explanations; however, explainable AI algorithms are not the only solution to improve system explainability (for example, designing effective explanation interfaces).

Local explanations

• Feature-importance analysis (e.g. random forest feature permutation analysis, saliency maps, feature reconstructions, individual condition expectation (ICE) plots)
• Partial dependence plots (PDPs)
• Shapley values

Global explanations

• Local interpretable model-agonistic explanations (LIME)
• Integrated gradients

Example based

Contrastive, counterfactual, data explorers/visualisation.

• What-If Tool
• Error Analysis

Model-agnostic methods

Feature-importance methods

• interpretML
• aix360
• SageMaker Clarify
• Captum
• SHAP

Specifically for neural-network interpretations

• Google’s Language Interpretability Tool (LIT)
• Captum

Specifically for deep learning in cloud environments

• SageMaker ML platform
• Google’s Explainable AI (xAI)

Advice on appropriate explanations is available in the NAIC’s Implementing Australia’s AI Ethics Principles report.

8. Contestability

8.1    Notification of AI affecting rights

You should notify individuals, groups, communities or businesses when an administrative action materially influenced by an AI system has a legal or similarly significant effect on them. This notification should state that the action was materially influenced by an AI system and include information on available review rights and whether and how the individual can challenge the action. 

An action producing a legal effect is when an individual, group, community or business’s legal status or rights are affected, and includes:

• provision of benefits granted by legislation
• contractual rights.

An action producing a similarly significant effect is when an individual, group, community or business’s circumstances, behaviours or choices are affected, and includes: 

• denial of consequential services or support, such as housing, insurance, education enrolment, criminal justice, employment opportunities and health care services
• provision of basic necessities, such as food and water.

A decision may be considered to have been materially influenced by an AI system if:

• the decision was automated by an AI system, with little to no human oversight
• a component of the decision was automated by an AI system, with little to no human oversight (for example, a computer makes the first 2 limbs of a decision, with the final limb made by a human)
• the AI system is likely to influence decisions that are made (for example, the output of the AI system recommended a decision to a human for consideration or provided substantive analysis to inform a decision). 

‘Administrative action’ is any of the following:

• making, refusing or failing to make a decision
• exercising, refusing or failing to exercise a power
• performing, refusing or failing to perform a function or duty.

Note: this guidance is designed to supplement, not replace, existing administrative law requirements pertaining to notification of administrative decisions. The Attorney-General’s Department is leading work to develop a consistent legislative framework for automated decision making (ADM), as part of the government’s response to recommendation 17.1 of the Robodebt Royal Commission Report. The Australian Government AI assurance framework will continue to evolve to ensure alignment as this work progresses.

8.2    Challenging administrative actions influenced by AI

Individuals, groups, communities or businesses subject to an administrative action materially influenced by an AI system that has a legal or similarly significant effect on them should be provided with an opportunity to challenge this action. This is an important administrative law principle. See guidance on section 8.1 above for assistance interpreting terminology.

Administrative actions may be subject to both merits review and judicial review. Merits review considers whether a decision made was the correct or preferable one in the circumstances, and includes internal review conducted by the agency and external review processes. Judicial review examines whether a decision was legally correct.

You should ensure that review rights that ordinarily apply to human-made decisions or actions are not impacted or limited because an AI system has been used.

Notifications discussed at section 8.1 should include information about available review mechanisms so that people can make informed decisions about disputing administrative actions.

You will need to ensure a person within your agency is able to answer questions in a court or tribunal about an administrative action taken by an AI system if that matter is ultimately challenged. Review mechanisms also impact on the obligation to provide reasons. For example, the Administrative Decisions (Judicial Review) Act 1977 gives applicants a right to reasons for administrative decisions.

9. Accountability

9.1    Establishing responsibilities

Establishing clear roles and responsibilities is essential for ensuring accountability in the development and use of AI systems. In this section, you are asked to identify the individuals responsible for 3 key aspects of your AI system:

Use of AI insights and decisions

The person responsible for the application of the AI system’s outputs, including making decisions or taking actions based on those outputs.

Monitoring the performance of the AI system

The person responsible for overseeing the ongoing performance and safety of the AI system, including monitoring for errors, biases or unintended consequences. 

Data governance

The person responsible for the governance of the data used for operating, training or validating the AI system. 

Where feasible, it is recommended that these 3 roles not all be held by the same person. The responsible officers should be appropriately senior, skilled and qualified for their respective roles. 

9.2    Training of AI system operators

AI system operators play a crucial role in ensuring the responsible and effective use of AI. They must have the necessary skills, knowledge and judgment to understand the system’s capabilities and limitations, how to appropriately use the system, interpret its outputs and make informed decisions based on those outputs.

In your answer, describe the process for ensuring AI system operators are adequately trained and skilled. This may include:

Initial training

What training do operators receive before being allowed to use the AI system? Does this training cover technical aspects of the system, as well as ethical and legal considerations?

Ongoing training

Is there a process for continuous learning and skill development? How are operators kept up to date with changes or updates to the AI system?

Evaluation

Are operators’ skills and knowledge assessed? Are there any certification or qualification requirements?

Support

What resources and support are available to operators if they have questions or encounter issues?

Consider whether this needs to be tailored to the specific needs and risks of your AI system or proposed use case or whether general AI training requirements are sufficient.

10. Human-centred values

10.1    Incorporating diversity

Diversity of perspective promotes inclusivity, mitigates biases, supports critical thinking and should be incorporated in all AI system lifecycle stages. 

AI systems require input from stakeholders from a variety of backgrounds, including different ethnicities, genders, ages, abilities and socio-economic statuses. This also includes people with diverse professional backgrounds, such as ethicists, social scientists and domain experts relevant to the AI application. Determining which stakeholders and user groups to consult, which data to use, and the optimal team composition will depend on your AI system. 

The following examples demonstrate the often-unintended negative consequences of AI systems that failed to adequately incorporate diversity into relevant lifecycle stages. 

AI systems ineffective at predicting recidivism outcomes for defendants of colour and underestimating the health needs of patients from marginalised racial and ethnic backgrounds.

AI job recruitment systems unfairly affecting employment outcomes.

Algorithms used to prioritise patients for high-risk care management programs were less likely to refer black patients than white patients with the same level of health.

An AI system designed to detect cancers had shown biases towards lighter skin tones stemming from an oversight in collecting a more diverse set of skin tone images, potentially delaying life-saving treatments.

Resources, including approaches, templates and methods to ensure sufficient diversity and inclusion of your AI system, are described in the NAIC’s Implementing Australia’s AI Ethics Principles report.

10.2    Human rights obligations

You should consult an appropriate source of legal advice or otherwise ensure that your AI use case and use of data align with human rights obligations. If you have not done so, explain your reasoning.

It is recommended that you complete this question after you have completed the previous sections of the assessment. This will provide more complete information to enable an assessment of the human rights implications of your AI use case.

In Australia, it is unlawful to discriminate on the basis of a number of protected attributes including age, disability, race, sex, intersex status, gender identity and sexual orientation in certain areas of public life, including education and employment. Australia's federal anti‑discrimination laws are contained in the following legislation:

• Age Discrimination Act 2004
• Disability Discrimination Act 1992
• Racial Discrimination Act 1975
• Sex Discrimination Act 1984.

Human rights are defined in the Human Rights (Parliamentary Scrutiny) Act 2011 as the rights and freedoms contained in the 7 core international human rights treaties to which Australia is a party, namely the: 

• International Covenant on Civil and Political Rights (ICCPR).
• International Covenant on Economic, Social and Cultural Rights (ICESCR).
• International Convention on the Elimination of All Forms of Racial Discrimination (CERD).
• Convention on the Elimination of All Forms of Discrimination against Women (CEDAW).
• Convention against Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment (CAT).
• Convention on the Rights of the Child (CRC).
• Convention on the Rights of Persons with Disabilities (CRPD).

11. Internal review and next steps

11.1    Legal review of AI use case

If the threshold assessment in section 3 results in a risk rating of ‘medium’ or ‘high’, your AI use case must undergo legal review to ensure that the use case and associated use of data meet legal requirements.

The nature of the legal review is context dependent. Without limiting the scope of legal review, examples of potentially applicable legislation, policies and frameworks are outlined at Attachment A of the Policy for the responsible use of AI in government.

If there are significant changes to the AI use case (including changes introduced due to recommendations from internal or external review), then the advice should be revisited to ensure the AI use case and associated use of data continues to meet legal requirements.

11.2    Risk summary table

To complete the risk summary table, list any: 

• risks assessed in section 3 (the threshold assessment) as ‘medium’ or ‘high’ 
• instances where you have answered ‘no’ to questions in sections 4 to 10. You are encouraged to identify risk treatments in relation to these, however, you do not need to assign a residual risk rating to those risks
• additional risks that have been identified throughout the assessment process 
• risk treatments identified during internal review (section 11.3) and, if applicable, external review (section 11.4) – using the risk matrix in section 3 to assess residual risk.

11.3    Internal review of AI use case

This requires an internal agency governance body designated by your agency’s Accountable Authority to review the assessment and the risks outlined in the risk summary table. 

The governance body may decide to accept any ‘medium’ risks, to recommend risk treatments, or decide not to accept the risk and recommend not proceeding with the AI use case. You should list the recommendations of your agency governance body in the text box provided.

11.4    External review of AI use case

If, following internal review (section 11.3), there are any residual risks with a ‘high’ risk rating, your agency should consider whether the AI use case and this assessment would benefit from external review. This external review may recommend further risk treatments or adjustments to the use case.

In line with the APS Strategic Commissioning Framework, consider whether someone in the APS could conduct this review or whether the nature of the use case and identified risks warrant independent outside review and expertise. 

Your agency must consider recommendations of an external review, decide which to implement, and whether to accept any residual risk and proceed with the use case. If applicable, you should list any recommendations arising from external review in the text box provided and record the agency's response to these recommendations.

Attachment

Risk consequence rating advice

Negatively affecting public accessibility or inclusivity of government services

Insignificant

• Insignificant compromises to accessibility or inclusivity of services.
• Minor technical issues causing brief inconvenience but no actual barriers to access or inclusion.
•  Issues rapidly resolved with minimal impact on user experience.

Minor

• Limited, reversable compromises to accessibility or inclusivity of services.
• Some people experience difficulties accessing services due to technical issues or design oversights.
• Barriers are short-term and addressed once identified, with additional support provided to people affected.

Moderate

• Many compromises are made to the accessibility or inclusivity of services.
• Considerable access challenges for a modest number of users.
• Resolving access issues requires substantial effort and resources.
• Certain groups may be disproportionately impacted.
• Affected users experience frustration and delays in receiving services.

Major

• Extensive compromises are made to the accessibility or inclusivity of services, may include some essential services.
• Ongoing delays that require external technical assistance to resolve.
• Widespread inconvenience, frustration, public distress and potential legal implications.
• Vulnerable user groups disproportionately impacted.

Severe

• Widespread irreversible ongoing compromises are made to the accessibility or inclusivity of services, including some essential services.
• Majority of users, especially vulnerable groups affected.
• Essential services inaccessible for extended periods, causing significant public distress, legal implications, and a loss of trust in government efficiency.
• Comprehensive and immediate actions are urgently needed to rectify the situation.

Unfair discrimination against individuals, communities or groups

Insignificant

• Negligible instances of discrimination, with virtually no discernible effect on individuals, communities, or groups.
• Issues are proactively identified and rapidly addressed before causing harm.

Minor

• Limited instances of unfair discrimination occur, affecting a small number of individuals.
• Relatively isolated cases, and corrective measures minimise their impact.

Moderate

• Moderate levels of discrimination leading to noticeable harm to certain individuals, communities, or groups.
•  These incidents raise bias and fairness concerns and require targeted interventions.

Major

• Significant discrimination results in major, tangible harm to individuals and multiple communities or groups.
• Rebuilding trust requires substantial reforms and remediation efforts.

Severe

• Pervasive and systemic discrimination causes severe harm across a broad spectrum of the population, particularly marginalised and vulnerable groups.
• Public outrage, potential legal action, and a profound loss of trust in government.
• Immediate, sweeping reforms and accountability measures are required.

Perpetuating stereotyping or demeaning representations of individuals, communities or groups

Insignificant

• Inadvertently reinforce mild stereotypes, but these instances are quickly identified and rectified with no lasting harm or public concern.
• Minor
• Isolated cases of stereotyping, affecting limited members of community with some noticing and raising concerns.
• Prompt action mitigates the issue, preventing broader impact.

Moderate

• Moderate stereotyping by AI systems leads to noticeable public discomfort and criticism.
• Disproportionally affecting certain communities or groups.
• Requires targeted corrective measures to address and prevent recurrence.

Major

• Significant and widespread reinforcement of harmful stereotypes and demeaning representations.
• Causes public outcry and damages the relationship between communities and government entities.
• Urgent, comprehensive strategies are needed to rectify these representations and restore trust.

Severe

• Pervasive and damaging stereotyping severely harms multiple communities, leading to widespread distress.
• Potential legal consequences, and a profound breach of trust in government use of technology.
• Requires immediate, sweeping actions to address the harm, including system overhauls and public apologies.

Harm to individuals, communities, groups, businesses or the environment

Insignificant

• Inconsequential glitches with no real harm to the public, business operations or ecosystems.
• Easily managed through routine measures.

Minor

• Isolated incidents mildly affecting the public.
• Slight inconveniences or disruptions to businesses, leading to manageable financial costs.
• Limited manageable environmental disturbances affecting local ecosystems or resource consumption.

Moderate

• Noticeable negative effects on the public.
• Businesses face operational challenges or financial losses, affecting their competitiveness.
• Obvious environmental degradation, including pollution or habitat disruption, prompting public concern.

Major

• Significant public harm causing distress and potentially lasting damage.
• Significant harm to a wide range of businesses, resulting in substantial financial losses, layoffs, and long-term reputational damage.
• Compromises ecosystem wellbeing causing substantial pollution, loss of biodiversity, and resource depletion.

Severe

• Widespread, profound harm and severe distress affecting broad segments of the public.
• Profound damage across the business sector, leading to bankruptcies, major job losses, and a lasting negative impact on the economy.
• Comprehensive environmental destruction, leading to critical loss of biodiversity, irreversible ecosystem damage, and severe resource scarcity.

Compromising privacy due to the sensitivity, amount or source of the data being used by an AI system

Insignificant

• Insignificant data handling errors occur without compromising sensitive information.
• Incidents are quickly rectified, maintaining public trust in data security.

Minor

• Isolated exposure of limited sensitive data affects a small group of individuals.
• Swift actions taken to secure the data and prevent further incidents.

Moderate

• Breach of moderate amounts of sensitive data, leading to privacy concerns among the affected populace.
• Some individuals experience inconvenience and distress.

Major

• Serious misuse of sensitive private data affects a large segment of the population, leading to widespread privacy violations and a loss of public trust.
• Comprehensive measures are urgently required to secure data and address the privacy breaches.

Severe

• Significant potential to expose sensitive information of a vast number of individuals, causing severe harm, identity-theft risks; use of sensitive personal information in a way that is likely to draw public criticism with limited ability for individuals to choose how their information is used.
• Significant potential to harm trust in government-information handling with potential for lasting consequences.

Raising security concerns due to the sensitivity or classification of the data being used by an AI system

Insignificant

• Inconsequential security lapses occur without actual misuse of sensitive data.
• Quickly identified and corrected with no real harm done.
• These types of incidents may serve as prompts for reviewing security protocols.

Minor

• A limited security breach involves unauthorised access to protected data affecting a small number of records with minimal impact.
• Immediate actions secure the breach, and affected individuals are notified and supported.
• Incident is catalyst for review of security protocols.

Moderate

• Security incident leads to the compromise of a moderate volume of sensitive data, raising concerns over data protection and privacy.
• The breach necessitates a thorough investigation, enhanced security measures.

Major

• A significant security breach results in extensive unauthorised access to sensitive or protected data, causing considerable concern and distress among the public.
• Urgent security upgrades and support measures for impacted individuals are implemented. to restore security and trust.

Severe

• A massive security breach exposes a vast amount of sensitive and protected data, leading to severe implications for national security, public safety, and individual privacy.
• This incident triggers an emergency response, including legal actions, a major overhaul of security systems, and long-term support for those affected.

Raising security concerns due to implementation, sourcing or characteristics of the AI system

Insignificant

• Inconsequential security concerns arise due to characteristics of the AI system, such as software bugs, which are promptly identified and fixed with no adverse effects on overall security.
• These issues may serve as lessons, leading to slight improvements in the system's security framework.

Minor

• Certain characteristics of the AI system lead to vulnerabilities that are exploited in a limited manner, causing minor security breaches.
• Immediate remediation measures are taken, and the system is updated to prevent similar issues.

Moderate

• A moderate security risk is realised when intrinsic features of the AI system allow for unintended access or data leaks.
• Incident affects a noticeable but contained component of the AI system.
• Prompts a comprehensive security review of the AI system and the implementation of more robust safeguards.

Major

• Significant security flaws in the AI system's design result in major breaches, compromising a large amount of data and severely affecting system integrity.
• Incident leads to an urgent overhaul of security measures and protocols, alongside efforts to mitigate the damage.

Severe

• Critical security vulnerabilities inherent to the AI system lead to widespread breaches, exposing vast quantities of sensitive data and jeopardising national security or public safety.
• The incident results in severe consequences, necessitating emergency responses, extensive system redesigns, and long-term efforts to recover from the breach and prevent recurrence.

Influencing decision-making affects individuals, communities, groups, businesses or the environment

Insignificant

• Decisions lead to negligible errors, swiftly identified and corrected with no harm to the public, business operations or the environment.
• Incidents may serve as learning opportunity for system improvement.

Minor

• Decisions result in minor inconveniences or errors affecting the public, business operations or finances or slight environmental impacts.
• All impacts reversible with prompt action.

Moderate

• Decisions cause moderate harm to the public, business operations or finances or noticeable environmental degradation.
• Targeted interventions are required to mitigate these effects.

Major

• Significant harm to the public, substantial business financial losses or operational disruptions, or significant environmental damage.
• Loss of confidence in government, operations, service delivery and partnerships.
• Significant harm to a wide range of businesses, resulting in substantial financial losses, layoffs, and long-term reputational damage.
• Compromises ecosystem wellbeing causing substantial pollution, loss of biodiversity, and resource depletion.

Severe

• AI's influence on critical decision-making processes leads to severe and widespread harm to public, business operations or finances or the environment.
• Potentially endangering lives or significantly impacting public safety, rights and trust.
• Causes massive job losses, undermining business economic stability and viability.
• Catastrophic loss of ecosystems, endangered species, and long-term ecological imbalance or severe resources depletion.

Posing a reputational risk or undermining public confidence in the government

Insignificant

• Isolated reputational issues arise, quickly addressed and explained.
• Causes negligible damage to public trust in government capabilities.

Minor

• Small-scale AI mishaps lead to brief public concern, slightly denting the government's reputation.
• Prompt clarification and corrective measures minimize long-term impact on public confidence
• Seen by the government as poor management.

Moderate

• Misapplications result in moderate public dissatisfaction and questioning of government oversight.
• Requires remedial actions to mend trust and address concerns.
• Seen by government and opposition as failed management.

Major

• Widespread public scepticism and criticism, majorly affecting the government's image.
• Requires substantial efforts to rebuild public confidence through transparency, accountability, and improvement of AI governance.
• High profile negative stories, seen by government and opposition as significant failed management.

Severe

• Severe misuse or failure of AI systems leads to profound public distrust and criticism.
• Significantly undermining confidence in government effectiveness and integrity.
• Requires comprehensive, long-term strategies for rehabilitation of public trust, including systemic changes and ongoing engagement.
• Seen by government and opposition as catastrophic failure of management.
• Minister expresses loss of confidence or trust in agency.

Risk likelihood table