Search

There was an inconsistent rollout of Copilot across agencies.

The experience and sentiment of trial participants may be affected by when their agency began participating in the trial and what version of Copilot their agency provided.

On the former, agencies received their Copilot licences between 1 January to 1 April 2024. Some agencies opted to distribute Copilot licences to participants later in this period once internal security assessments were complete. This meant that agencies participating in the trial had different timeframes to build capability and identify Copilot use cases, which could potentially affect participants’ overall sentiment and experience with Copilot. 

Further, agencies who joined the trial later may not have been able to contribute to early evaluation activities, such as the pre-use survey or initial interviews, therefore excluding their perspective and preventing later comparison of outcomes.

On the latter, since the trial began, Microsoft has released 60 updates for Copilot to enable new features – including rectifying early technical glitches. Due to either information security requirements or a misalignment between agency update schedules, the new features of Copilot may have been inconsistently adopted across participating agencies or at times, not at all. 

This means that there could be significant variability with Copilot functionality across trial participants and it is difficult for the evaluation to discern the extent to which participant sentiments are due to specific agency settings or Copilot itself.

Trial participants expressed a level of evaluation fatigue.

Agencies were encouraged to undertake their own evaluations to ensure the future adoption of Copilot or generative AI reflected their agency’s needs. Many participating agencies conducted internal evaluations of Copilot that involved surveys, interviews and productivity studies. 

Decreasing rates of evaluation activity participation over the trial indicates that trial participants may have become fatigued from evaluation activities. The survey response rate progressively decreased across the pre-use to pulse to post-use surveys. Lower response rates in the post-use survey (n = 831) and for those who completed both the pre-use and post-use survey (n = 330) may impact how representative the data is of the trial population. Participation in the Nous-facilitated focus groups and the post-use survey was impacted by these parallel initiatives and the subsequent evaluation fatigue. 

This means that the evaluation may not have been able to engage with a wide range of trial participants with a proportion of trial participants opting to only provide responses to their own agency evaluation. This may have been mitigated to a degree with some agencies sharing their results to the evaluation.

The impact of Copilot relied on trial participants’ self-assessment of productivity benefits.

The evaluation methodology relies on the trial participants’ self-assessed impacts of Copilot which may naturally under or overestimate impacts – particularly time savings. Where possible, the evaluation has compared its productivity findings against other APS agency evaluations and external research to verify the productivity savings put forth by trial participants. 

Nevertheless, there is a risk that the impact of Copilot – in particular the productivity estimates from Copilot use, may not accurately reflect Copilot’s actual productivity impacts.

2.1 Problem definition

Clearly and concisely identify the problem you are trying to solve. Use 100 words or less.

2.2 AI use case purpose

Clearly and concisely describe the purpose of your use of AI, focusing on how it will address the problem you have identified. Use 200 words or less.

2.3 Non-AI alternatives

Briefly outline nonAI alternatives that could address this problem. Use 100 words or less.  

2.4 Identifying stakeholders

Identify stakeholder groups that may be affected by the AI use case and briefly describe how they may be affected, whether positively or negatively. This will guide your consideration of expected benefits and potential risks in this assessment.  

Consider holding a brainstorm or workshop to help identify affected stakeholders and how they may be affected. A discussion prompt is provided in the guidance document.

2.5 Expected benefits

Considering the stakeholders identified in the previous question, identify the expected benefits of the AI use case. This should be supported by quantitative and/or qualitative analysis.  

Qualitative analysis should consider whether there is an expected positive outcome and whether AI is a good fit to accomplish the relevant task, particularly compared to nonAI alternatives identified. Benefits may include gaining new insights or data.

Consult the guidance document for resources to assist you. Aim for 300 words or less. 

The Pilot Australian Government artificial intelligence (AI) assurance framework (the framework) guides Australian Government agencies through impact assessment of AI use cases against Australia's AI Ethics Principles. It is intended to complement and strengthen – not duplicate – existing frameworks, legislation and practices that touch on government’s use of AI.

The draft framework should be read and applied alongside the Policy for the responsible use of AI in government and existing frameworks and laws to ensure agencies are meeting all their current obligations. Above all, Australian Government agencies must ensure their use of AI is lawful, constitutional and consistent with Australia’s human rights obligations and reflect this in the planning, design and implementation of AI use cases from the outset. 

Assurance is an essential part of the broader governance of government AI use. In June 2024, the Australian Government and all state and territory governments endorsed the National framework for the assurance of artificial intelligence. The national framework establishes a nationally consistent, principles-based approach to AI assurance, that places the rights, wellbeing and interests of people first. By committing to these principles, governments are seeking to secure public confidence and trust that their use of AI is safe and responsible.  

This pilot assurance framework is exploring mechanisms to support Australian Government implementation of these nationally agreed principles. Evidence gathered through the pilot will inform the DTA’s recommendations to government on future AI assurance mechanisms, as part of next steps for the Policy for the responsible use of AI in government.

The framework will continue to evolve over time. Please email the Digital Transformation Agency (DTA) at aistandards@dta.gov.au if you have any questions regarding the framework.

AI use cases covered by the framework

For the purposes of the framework, agencies should apply the Organisation for Economic Co‑operation and Development (OECD) definition of AI:

What happens at this state

Through the Data and Digital Government Strategy and a suite of digital policies, the DTA articulates the government’s vision for its digital enterprise and provides guardrails to assist agencies with digital delivery in-line with the government’s whole of economy strategic direction.

The Strategic Planning state provides a structure for agencies to complete short, medium and long-term planning of digital and ICT enabled projects through their Digital Investment Plans (DIPs). Through the DIP process, agencies can identify capability gaps and identify opportunities to promote reuse and shared benefit. Insights on future investment needs that should be considered as part of the prioritisation of shorter-term investment decisions are passed on to the Prioritisation state. The longer-term view provided through Strategic Planning also informs the APS’s data, digital and cyber workforce capability and capacity needs.

Why it is important

Strategic Planning supports the government to build a mature digital capability to deliver its digital and ICT objectives. By working closely with government agencies, the DTA can present government with a clear understanding of current and likely future investments to help identify key digital and ICT capability gaps.

By having an agreed digital and ICT architecture and a shared view of common services and opportunities for reuse of digital and ICT solutions across the Australian Public Service (APS), we can provide government with a clearer pathway to deliver its digital transformation.

Strategic Planning ensures the government’s investments in digital and ICT balances whole-of-government strategic digital and ICT priorities with individual agency outcomes and business needs.

What agencies need to do

Through the Data and Digital Government Strategy and a suite of digital policies, the DTA articulates the government’s vision for its digital enterprise and provides guardrails to assist agencies with digital delivery in-line with the government’s whole of economy strategic direction.

The Strategic Planning state provides a structure for agencies to complete short, medium and long-term planning of digital and ICT enabled projects through their Digital Investment Plans (DIPs). Through the DIP process, agencies can identify capability gaps and identify opportunities to promote reuse and shared benefit. Insights on future investment needs that should be considered as part of the prioritisation of shorter-term investment decisions are passed on to the Prioritisation state. The longer-term view provided through Strategic Planning also informs the APS’s data, digital and cyber workforce capability and capacity needs.

What the DTA will do

• Provide strategic advice to government to drive the transformation of Public Sector digital investments that effectively deliver services and outcomes for the benefit of all Australians, including delivering on the Data and Digital Government Strategy vision.
• Support agencies to align their enterprise-level digital strategies, plans and investments with the Data and Digital Government Strategy.
• Provide templates and guidance to agencies on the development of longer-term digital investment plans.
• Work in collaboration with the Australian Public Service Commission to increase APS digital and data capabilities through the Data, Digital and Cyber Workforce plan.
• Use the Australian Government Architecture (AGA) to guide agency planning activities. In tandem with the government’s digital policies, the AGA helps to drive standardisation, increases the reuse of common capabilities and supports the effective design of an investment in modern digital capabilities. The AGA also feeds into the prioritisation and contestability stages of the IOF.
• Support key forums including the Secretaries Digital and Data Committee, the Digital Leadership Committee, and the Australian Government Architecture Content Advisory Forum to support and drive the outcomes of the Data and Digital Government Strategy.