Search

Statement 8: Apply watermarking techniques

AI watermarking can be used to embed visual or hidden markers into generated content, so that its creation details can be identified. It provides transparency, authenticity, and trust to content consumers.

Visual watermarks or disclosures provide a simple way for someone to know they are viewing content created by, or interacting with, an AI system. This may include generated media content or GenAI systems.

The Coalition for Content Provenance and Authenticity (C2PA)  is developing an open technical standard for publishers, creators, and consumers to establish the origin and edits of digital content. Advice on the use of C2PA is out of scope for the standard.

Agencies must:

• Criterion 22: Apply visual watermarks and metadata to generated media content to provide transparency and provenance, including authorship.

  This will only apply where AI generated content may directly impact a user. For instance, using AI to generate a team logo would not need to be watermarked.

• Criterion 23: Apply watermarks and metadata that are WCAG compatible where relevant

• Criterion 24: Apply visual and accessible content to indicate when a user is interacting with an AI system.
  For example, this may include adding text to a GenAI interface so that users are aware they are interacting with an AI system rather than a human.

Agencies should:

• Criterion 25: For hidden watermarks, use watermarking tools based on the use case and content risk.

  This includes:

  • including provenance and authorship information
  • encrypting watermarks for high-risk content
  • using an existing tool or technique when practicable
  • embedding watermarks at the AI training stage to improve their effectiveness and allows additional information such as content modification to be included
  • verifying that the watermark does not impact the quality or efficiency of content generation, such as image degradation or text readability
  • including data sources, such as publicly available content used for AI training to manage copyright risks, and product details such as versioning information.

• Criterion 26: Assess watermarking risks and limitations.

  This includes:

  • ensuring users understand there is a risk of third parties replicating a visual watermark and to not over rely on watermarks, such as sourcing content from external sources
  • preventing third-party use of watermarking algorithms to create their own content and act as the original content creator
  • consider situations where watermarking is not beneficial. For example, watermarking can be visually distracting for decision makers, or when its overused in low-risk applications
  • consider situations where malicious actors might remove or replicate the watermark to reproduce content generated by AI
  • managing copyright or trademark risks related to externally sourced data.

Statement 9: Conduct pre-work

Agencies must:

• Criterion 27: Define the problem to be solved, its context, intended use, and impacted stakeholders.

  This includes:

  • analysing the problem through problem-solving frameworks such as root cause analysis, design thinking, and DMAIC (define, measure, analyse, improve, control)
  • define user needs, system goals and the scope of AI in the system
  • identifying and documenting stakeholders, including:
    • internal or external end-users, such as APS staff or members of the public
    • indigenous Australians, refer to Framework for Governance of Indigenous Data
    • people with lived experiences, including those defined by religion, ethnicity, or migration status
    • data experts, such as owners of the data being used to train and validate the AI system
    • subject matter experts, such as internal staff
    • the development team, including SROs, architects, and engineers.
  • understanding the context of the problem such as interacting processes, data, systems, and the internal and external operating environment
  • phrasing the problem in a way that is technology agnostic.

• Criterion 28: Assess AI and non-AI alternatives.

  This includes:

  • starting with the simplest design, experimenting, and iterating
  • validate and justify the need for AI by conducting an objective quality evidence assessment
  • differentiating parts that could be solved by traditional software from parts that could benefit from AI
  • determine why using AI would be more beneficial over non-AI alternatives by comparing KPIs
  • considering the interaction of any AI and non-AI components
  • considering existing agency solutions, commercial, or open-source off-the-shelf products
  • examining capabilities, performance, cost, and limitations of each option
  • conducting proof of concept and pilots to assess and validate the feasibility of each option
  • for transformative use cases, consider foundation and frontier models. Foundation models are quite versatile, trained on large data sets, and can be fine-tuned for specific contexts. Frontier models are at the forefront of AI research and development, trained on extensive datasets, and may demonstrate creativity or reasoning.

• Criterion 29: Assess environmental impact and sustainability. 

  Developing and using AI systems may have corresponding trade-offs with electricity usage, water consumption, and carbon emissions.

• Criterion 30: Perform cost analysis across all aspects of the AI system.

  This includes:

  • infrastructure, software, and tooling costs for:
    • acquiring and processing data for training, validation, and testing
    • tuning the AI system to your particular use case and environment
    • internally or externally hosting the AI system
    • operating, monitoring, and maintaining the AI system.
  • cost of human resources with the necessary AI skills and expertise.

• Criterion 31: Analyse how the use of AI will impact the solution and its delivery.

  This includes:

  • identifying the type of AI and classification of data required
  • identifying the implications of integrating the AI system with existing departmental systems and data, or as a standalone system
  • identifying legislation, regulations, and policies.

Statement 10: Adopt a human-centred approach

Agencies must

• Criterion 32: Identify human values requirements.

  Human values represent what people deem important in life such as autonomy, simplicity, tradition, achievement, and social recognition.

  This includes:

  • using traditional requirement elicitation techniques such as surveys, interviews, group discussions and workshops to capture relevant human values for the AI use case
  • translating human values into technical requirements, which may vary depending on the risk level and AI use case
  • reviewing feedback to identify ignored human-values in the AI system
  • understanding the hierarchy of human values and emphasising those with higher relevance
  • considering social, economic, political, ethical, and legal values when designing AI systems
  • considering human values that are domain specific and based on the context of the AI system.

• Criterion 33: Establish a mechanism to inform users of AI interactions and output, as part of transparency.

  Depending on use case this may include:

  • incorporating visual cues on the AI product when applicable
  • informing users when text, audio or visual messages addressed to them are generated by AI
  • including visual watermarks to identify content generated by AI
  • providing transparency on whether a user is interacting with a person, or system
  • including a disclaimer on the limitations of the system
  • displaying the relevance and currency of the information being provided
  • persona level transparency adhering to need-to-know principles
  • providing alternate channels where a user chooses not to use the AI system. This may include channels such as a non-AI digital interface, telephony, or paper.

• Criterion 34: Design AI systems to be inclusive, ethical, and meets accessibility standards using appropriate mechanisms. 

  This includes:

  • identifying affirmative actions or preferential treatment that apply for any person or specific stakeholder groups
  • ensuring diversity and inclusion requirements, and guidelines, are met throughout the entire AI lifecycle
  • providing justification to situations such as pro-social policy outcomes
  • reviewing and revisiting ethical considerations throughout the AI system lifecycle.

• Criterion 35: Define feedback mechanisms.

  This includes:

  • providing options to users on the type of feedback method they prefer
  • providing users with the choice to dismiss feedback
  • provide the user with the option to opt-out of the AI system
  • ensuring measures to protect personal information and user privacy
  • capturing implicit feedback to reflect user's preferences and interactions, such as accepting or rejecting recommendations, usage time, or login frequency
  • capturing explicit feedback via surveys, comments, ratings, or written feedback.

• Criterion 36: Define human oversight and control mechanisms.

  This includes:

  • identifying conditions and situations that need to be supervised and monitored by a human, conditions that need to be escalated by the system to a supervisor or operator for further review and approval, and conditions that should trigger transfer of control from the AI system to a supervisor or operator
  • defining the system states, errors, and other relevant information that should be observable and comprehensible to an informed human
  • defining the pathway for the timely intervention, decision override, or auditable system takeover by authorised internal users
  • subsets of inputs and outputs that may result in harm should be recorded for monitoring, auditing, contesting, or validation. This will facilitate reviewing of false positives against inputs that triggered them, and of false negatives that result in harms
  • identifying situations where a supervising human might become disengaged and designing the system to attract the operators attention
  • map human oversight and control requirements to corresponding risks they mitigate
  • identifying required personas and defining their roles
  • adherence to privacy and security need-to-know principles.

Agencies should:

• Criterion 37: Involve users in the design process.

  The intention is to promote better outcomes for managing inclusion and accessibility by setting expectations at the beginning of the AI system lifecycle.

  This includes:

  • considering security guidance and the need-to-know principle
  • involving users in defining requirements, evaluating, and trialling systems or products.
     

Statement 11: Design safety systemically

Agencies must:

• Criterion 38: Analyse and assess harms.

  This includes:

  • Utilising functional safety standards that provide frameworks for a systematic and robust harms analysis.

• Criterion 39: Mitigate harms by embedding mechanisms for prevention, detection, and intervention.

  This includes:

  • designing the system to avoid the sources of harm
  • designing the system to detect the sources of harm
  • designing the system to check and filter its inputs and outputs for harm
  • designing the system to check for sensitive information disclosure
  • designing the system to monitor faults in its operation
  • designing the system with redundancy
  • designing intervention mechanisms such as warnings to users and operators, automatic recovery to a safe state, transfer of control, and manual override
  • designing the system to log the harms and faults it detects
  • designing the system to disengage safely as per requirements
  • for physical systems, designing proper protective equipment and procedures for safe handling
  • ensuring the system meets privacy security requirements and adheres to the need-to-know principle for information security.

Agencies should

• Criterion 40: Design the system to allow calibration at deployment.

  This includes:

  • where initial setup parameters are critical to the performance, reliability, and safety of the AI system.
     

Statement 12: Define success criteria

Agencies must

• Criterion 41: Identify, assess, and select metrics appropriate to the AI system.

  Relying on a single metric could lead to false confidence, while tracking irrelevant metrics could lead to false incidents. To mitigate these risks, analyse the capabilities and limitations of each metric, select multiple complementary metrics, and implement methods to test assumptions and to find missing information.

  Considerations for metrics includes:

  • value-proposition metrics – benefits realisation, social outcomes, financial measures, or productivity measures
  • performance metrics – precision and recall for classification models, mean absolute error for regression models, or bilingual evaluation understudy (BLEU) for text generation. This can include summarisation tasks, inception score for image generation models, or mean opinion score for audio generation
  • training data metrics – data diversity and data quality related measures
  • bias-related metrics – demographic parity to measure group fairness, fairness through awareness to measure individual fairness, counterfactual fairness to measure causality-based fairness
  • safety metrics – likelihood of harmful outputs, adversarial robustness, or potential data leakage measures
  • reliability metrics – availability, latency, mean time between failures (MTBF), mean time to failure (MTTF), or response time
  • citation metrics – measures related to proper acknowledgement and references to direct content and specialised ideas
  • adoption-related metrics – adoption rate, frequency of use, daily active users, session length, abandonment rate, or sentiment analysis
  • human-machine teaming metrics – total time or effort taken to complete a task, reaction time when human control is needed, or number of times human intervention is needed
  • qualitative measures – checking the well-being of the humans operating or using the AI system, or interviewing participants and observing them while using the AI system to identify usability issues
  • drift in AI system inputs and outputs - changes in input distribution, outputs, and performance over time.
     

• After metrics have been identified, understand and assess the trade-offs between the metrics.

  This includes:

  • assessing trade-offs between different success criteria
  • determining the possible harms with incorrect output, such as a false positive or false negative
  • analysing how the output of the AI system could be used. For example, determine which instance would have greater consequences: a false negative that would fail to detect a cyberattack; or a false positive that incorrectly flags a legitimate user as a threat
  • assessing the trade-offs among the performance metrics
  • understanding the trade-offs with costs, explainability, reliability, and safety
  • understanding the limitations of the selected metric and ensure measures are considered when building the AI system, such as selecting data and training methods
  • trade-offs are documented, understood by stakeholders, and accounted for in selecting AI models and systems
  • optimising the metrics appropriate to the use case.

Agencies should

• Criterion 42: Reevaluate the selection of appropriate success metrics as the AI system moves through the AI lifecycle.

• Criterion 43: Continuously verify correctness of the metrics.

  Before relying on the metrics, verify the following:

  • metrics are accurately reflected when the AI system does not have enough information
  • metrics correctly reflect errors, failures, and successful task performance.

Statement 13: Establish data supply chain management processes

Agencies must

• Criterion 44: Create and collect data for the AI system and identify the purpose for its use.

  It is important to identify:

  • what data will be used and is fit-for-purpose for the AI system
  • the sensitivity of the data, such as personal, protected, or otherwise sensitive
  • consent provided on usage including when to retain or destroy data, ensuring the proposed uses in the AI system align with the original limits of the consent
  • speed and mode of the data supply
  • how the data will be used at each stage of the AI system
  • where the data will be stored at each stage of the AI system
  • changes to the data at different points of the AI system
  • methods to manage and monitor data access
  • methods to manage any real-time data changes
  • data retention policies
  • cross-agency or cross-border data governance, if relevant
  • any risks and challenges associated with data elements of off-the-shelf AI models, products, or services in the AI system
  • cyber supply chain management
  • data quality monitoring and remediation
  • comprehensive documentation at each stage of the AI system to facilitate traceability and accountability
  • adherence to relevant legislation.
  • The consent framework for use of data across the AI system should satisfy the following:
    • clear framework
    • kept up to date
    • individuals are provided with informed consent for how their data will be used
    • a dedicated team to own and maintain a register on how data is being used and to show compliance with the terms of the consent
  • The data should be thought of in groupings or packages, including:
    • the data within the organisation
    • the data surrounding the algorithm, APIs, and user interface
    • the data used to train the AI system
    • the data used for testing and integration
    • data inputted at regular intervals in monitoring the data
    • the data used at deployment, including input and output data from and to users.

• Criterion 45: Plan for data archival and destruction.

  Consider the following: 

  • will data be made available for future use, and what data
  • restrictions and access controls in place
  • will data be restricted until a specific date
  • file formats to ensure data remains available during the archival period
  • alignment with data sharing arrangements
  • arrangements for data used to train and test AI models, and associated model management arrangements
  • clear criteria for data archival and destruction for the data used at each stage of the AI lifecycle
  • guidelines in the Information management for records created using Artificial Intelligence (AI) technologies | naa.gov.au.

Agencies should:

• Criterion 46: Analyse data for use by mapping the data supply chain and ensuring traceability.

  Mapping the data supply chain to the AI system involves capturing how data will be stored, shared, and processed, particularly at the training and testing stages, which involve regular injections of data. When mapping the data account for:

  • how data was sourced
  • what data is required by the system, ensuring that excess data or data irrelevant to the functioning of system is not consumed by the system
  • the amount and type of data the system will use
  • what could affect the reliable accessibility of data
  • how data will be fused and transformed
  • how will the data be secured at rest and in transit
  • how will the data be used by the system.

  Ensuring traceability entails maintaining awareness of the flow of data across the AI system.

  This includes:

  • data sovereignty controls and considerations including legal implications for geographic locations for data (including its metadata and logs) when at rest, in transit, or in use. For classified data processing on cloud platforms, it is recommended to use cloud service providers and cloud services located in Australia, as per Cloud assessment and authorisation | Cyber.gov.au
  • providing the level of detail for debugging data errors and troubleshooting
  • enforcing organisational policies on information management
  • enhancing visibility over changes to the data occurring during migrations, system updates, or other errors
  • supporting users to identify and fix data issues with a clear information audit trail
  • supporting diagnosis for bias
  • managing the quality of data to maintain availability and consistency. 

• Criterion 47: Implement practices to maintain and reuse data.

  This involves determining ongoing mechanisms for ensuring data is protected, accessible, and available for use in line with the original consent parameters.

  Any changes in data scope, including expansion in scope and usage patterns, would need to be monitored and addressed.

Statement 14: Implement data orchestration processes