Technical standard for government’s use of artificial intelligence: Whole of AI lifecycle statement

Statement 6: Manage system bias 

Management of bias and its potential harms of an AI system is critical to ensuring compliance with federal anti-discrimination legislation. Australia’s anti-discrimination law states: 

…it is unlawful to discriminate on the basis of a number of protected attributes including age, disability, race, sex, intersex status, gender identity and sexual orientation in certain areas of public life, including education and employment. 

Certain forms of bias, such as affirmative measures for disadvantaged or vulnerable groups, play a constructive role in aligning AI systems to human values, intentions, and ethical principles. At the same time, it’s important to identify and address biases that may lead to unintended or harmful consequences. A balanced approach to bias management ensures that beneficial biases are preserved while minimising the impact of problematic ones.

When integrating off-the-shelf AI products, it’s essential to ensure they deliver fair and equitable outcomes in the targe operating environment. Conducting thorough bias evaluations becomes especially important when documentation or supporting evidence is limited.

Agencies must:

• Criterion 15: Identify how bias could affect people, processes, data, and technologies involved in the AI system lifecycle.

  Systemic biases: are rooted in societal and organisational culture, procedures, or practices that disadvantage or benefit specific cohorts. These biases manifest in datasets and in the processes throughout the AI lifecycle.

  Human bias: can affect design decisions, data collection, labelling, test selection, or any process that require judgment throughout the AI lifecycle. They could be conscious (implicit) or unconscious (explicit).

  Statistical and computational bias occurs when data used to train an AI system is not representative of the population. This is explored in more depth in the data section.

  This includes: 

  • establishing a bias management plan, outlining how bias will be identified, assessed, and managed across the AI system lifecycle
  • checking for systemic bias, which are rooted in societal and organisational culture, procedures, and practices that disadvantage or benefit specific cohorts. These biases manifest in datasets and in the processes throughout the AI lifecycle
  • checking for algorithmic bias in decision-making systems, where an output from an AI system might produce incorrect, unfair or unjustified results
  • checking for human bias, which can be conscious or unconscious biases in design decisions, data collection, labelling, test selection, or any process that requires judgment throughout the AI lifecycle
  • checking for statistical and computational bias, which can occur when data used to train an AI system is not representative of the population
  • checking for bias based on the application of AI, such identifying cognitive bias in a computer vision system
  • considering intended bias, such as identifying specific circumstances for a person or a group
  • considering inherent bias when reusing pre-trained AI models.
     
  • Examples of sources of bias includes:
    • Cognitive bias – systematic human inclinations or reasoning, such as subconscious judgements based on the current norms of individuals. Based on how people interpret and understand information in their surroundings, such as only using data that reinforces an individual's belief
    • Authority bias – tendency to provide greater weighting or consideration of information from an authority source
    • Availability bias – providing undue weighting to information or processes that they are, or have been, actively involved with
    • Confirmation bias – tendency to interpret, favour, or seek out information that reinforces a personal belief, value or understanding, such as a political alignment
    • Contextual bias – reliance upon unnecessary or irrelevant information which may unduly influence a decision
    • In-group or labelling bias – preferential treatment is provided to those who belong in the same group. Adversely, out-group bias is where unfavourable treatment is provided to those who belong in other groups
    • Stereotype bias – generalisations about an individual or group of people based on shared characteristics, such as age, gender, or ethnicity
    • Anchoring bias – tendency to rely heavily on the first piece of information they receive
    • Group think – tendency for people to strive for consensus within a group
    • Automation bias – tendency to rely on automated systems and ignore contradictory information made without automation.

• Criterion 16: Assess the impact of bias on your use case.

  This typically involves:

  • identifying stakeholders and the potential harms to them
  • identifying existing countermeasures and assessing their effectiveness
  • engaging with diverse and multi-disciplinary stakeholders in assessing the potential impacts of bias
  • using bias assessment tools relevant to your use case.

• Criterion 17: Manage identified bias across the AI system lifecycle.

  For off-the-shelf products, AI deployers should ensure that the AI system provides fair outcomes. Evaluating for bias will be critical where insufficient documentation from the off-the-shelf AI model supplier is provided.

  This involves: 

  • engaging multi-disciplinary skillsets and diverse perspectives, including:
    • policy owners, legal, architecture, data, IT experts, program managers, service delivery professionals, subject matter experts
    • people with lived experience, for example people with disability, gender or sexual diversity and people who are culturally and linguistically diverse.
  • implementing multiple approaches to reduce automation bias and monitor to detect unwanted bias that might emerge
  • identifying bias-specific documentation requirements such as data and model provenance records:
    • document selection criteria for selecting stakeholders, metrics, and other design-related decisions
    • document any discarded requirement, design, data, model, or tests with corresponding rationale
    • document biases that resulted in decommissioning the data, the model, the application, or the system
  • performing periodic context-based bias awareness training for teams
  • consideration of lifecycle stage-specific mitigations, including:
    • identify and validate root causes of bias before addressing them
    • identify corrective and preventive actions corresponding to the root causes of bias
    • identify fairness metrics at design. Performance metrics, such as accuracy and precision, aggregated over the entire dataset could hide bias. For example, a cancer detecting device with 90 per cent accuracy averaged across the entire dataset could hide underperformance on a minority population. Disaggregating performance metrics into suitable attributes can detect whether a system performs fairly across demographics, environmental conditions, and other risk factors
    • analyse data for bias and fix issues in the data. See Model and Context dataset section  for more information
    • test independence strategy, functional performance testing, fairness testing, and user acceptance testing
    • configure, calibrate, and monitor bias-related metrics during phased roll-out
    • monitor bias-related metrics and unintended consequences during operations. Provide mechanisms for end-users to report and escalate experiences of bias.
    • audit for how risks of bias are identified, assessed, and mitigated throughout the lifecycle.
    • find and use suitable tools that discover and test for unwarranted associations between the AI system outputs and protected input features
    • implement bias mitigation techniques after harmful bias has been identified
    • implement bias mitigation thresholds that can be configured post-deployment to ensure equity for cohorts, such as people with lived experience.