-
Executive summary
In the few years since its public introduction, generative AI (artificial intelligence) has become available and accessible to millions of people. The growing availability and speed of uptake in publicly available tools, such as ChatGPT, meant the Australian Public Service (APS) had to respond quickly to allow its workforce to experiment with generative AI in a safe, responsible and integrated way. To facilitate this experimentation, an appropriate generative AI tool needed to be selected.
Microsoft 365 Copilot (formerly Copilot for Microsoft 365) was one of the solutions available to enable the APS to undertake safe and responsible generative AI experimentation. On 16 November 2023, the Australian Government announced a 6-month whole-of-government trial of Copilot.
This decision was predicated on how swiftly and seamlessly Copilot, as a capability nested within existing whole-of-government contracting arrangements with Microsoft, could be deployed for rapid APS experimentation purposes. Further, as Copilot is a supplementary product that integrates with the existing applications within the Microsoft 365 suite, it also allowed staff to experiment and learn within the context of applications that were already familiar to them.
The trial involved the distribution of over 5,765 Copilot licenses between January to June 2024. The trial was non-randomised, with agencies nominating staff to be allocated a license. Trial participants comprised a range of APS classifications, job families, experience levels with generative AI, and expectations of generative AI capabilities.
Further detail on the background of the evaluation can be found in Appendix A.
More broadly, this trial and evaluation tested the extent to which much of the wider promise of generative AI capabilities would translate into real-world adoption by workers. The results will help the government consider future opportunities and challenges related to adopting generative AI.
This was just the first trial of a generative AI tool within the Australian Government and the future brings exciting opportunities to understand what other tools are available to explore a broad landscape of use cases.
Overview of the evaluation
Nous Group (Nous) was engaged by the Digital Transformation Agency (DTA) to assist with the evaluation of the trial. The Australian Centre of Evaluation was consulted on methodology and approach to ensure best practise. The evaluation was guided by 4 objectives and key lines of enquiry (KLEs) outlined in the table below.
Table 1. Evaluation objectives and key lines of enquiry Evaluation objective Key lines of enquiry Employee related outcomes Determine whether Copilot, as an example of generative AI, benefits APS productivity in terms of efficiency, output quality, process improvements and agency ability to deliver on priorities. What are the perceived effects of Copilot on APS employees? Productivity Evaluate APS staff sentiment about the use of Copilot. What are the perceived productivity benefits of Copilot? Whole-of-government adoption of generative AI Determine whether and to what extent Copilot, as an example of generative AI, can be implemented in a safe and responsible way across government. What are the identified adoption challenges of Copilot, as an example of generative AI, in the APS in the short and long term? Unintended outcomes Identify and understand unintended benefits, consequences, or challenges of implementing Copilot as an example of generative AI and the implications on adoption of generative AI in the APS. Are there any perceived unintended outcomes from the adoption of Copilot?
Are there broader generative AI effects on the APS?
The findings of the evaluation and the resulting implications are outlined at a high level in this executive summary. Further details are provided in the body of the report.
-
Glossary
Term Meaning AI in Government Taskforce Co-led by the DTA and the Department of Industry, Science and Resources (DISR), the AI in Government Taskforce aimed to deliver policies, standards, and guidance for the safe, ethical and responsible use of AI technologies by government. Confidence interval A confidence interval is a statistical concept used to estimate a population parameter based on sample data. It provides a range of values that likely contain the true population parameter with a certain level of confidence. Generative AI Generative AI is a branch of artificial intelligence focused on designing algorithms that generate novel outputs, such as text, images or sounds, based on learned patterns from data. Hallucinations Large Language Models (LLMs) such as Microsoft 365 Copilot are trained to predict patterns rather than understand facts, sometimes leading to it returning plausible sounding but inaccurate information, which is referred to as a ‘hallucination’. Large Language Model (LLM) Large language models are a category of foundation models trained on immense amounts of data making them capable of understanding and generating natural language and other types of content to perform a wide range of tasks. Microsoft 365 A cloud-based suite of productivity and collaboration tools offered by Microsoft, including Office applications, email and other services. Microsoft Office A suite of desktop productivity applications from Microsoft, including Word, Excel, PowerPoint and others. Microsoft Graph A Microsoft application programming interface (API) that provides access to data and intelligence across Microsoft 365 services, enabling developers to build apps that interact with organisational data. Mixed methods Combined use of both qualitative and quantitative research approaches to provide a more comprehensive understanding of the subject being evaluated. Microsoft 365 Copilot AI-enabled functionality embedded into the Microsoft 365 application suite. Formerly called Copilot for Microsoft 365. P-value A statistical measure that indicates the probability of observing a result as extreme as, or more extreme than, the one observed, assuming the null hypothesis is true. Trial participant An Australian Public Service staff member who participated in the whole-of-government Microsoft 365 Copilot trial, between January and July 2024. T-test A statistical test used to compare the means of 2 groups to determine if they are significantly different from each other, accounting for the variability in the data and sample size. -
Employee-related outcomes
-
Productivity
-
Employee-related outcomes
-
Productivity
-
Whole-of-government adoption of generative AI
-
Unintended outcomes
-
Whole-of-government adoption of generative AI
-
Unintended outcomes
-
Appendix A
-
Appendix A: Background
-
Appendix B
-
Appendix B: Methodology
-
Appendix C
-
Appendix C: Agency participation in the evaluation
C1. Overall participation
Portfolio Entity Agriculture, Fisheries and Forestry Department of Agriculture, Fisheries and Forestry
Grains Research and Development Corporation
Regional Investment Corporation
Rural Industries Research and Development (trading as AgriFutures Australia)
Attorney-General’s Australian Criminal Intelligence Commission
Australian Federal Police
Australian Financial Security Authority
Office of the Commonwealth Ombudsman
Climate Change, Energy, the Environment and Water Australian Institute of Marine Science
Australian Renewable Energy Agency
Department of Climate Change, Energy, Environment and Water
Bureau of Meteorology
Education Australian Research Council
Department of Education
Tertiary Education Quality and Standards Agency
Employment and Workplace Relations Comcare
Department of Employment and Workplace Relations
Fair Work Commission
Finance Commonwealth Superannuation Corporation
Department of Finance
Digital Transformation Agency
Foreign and Trade Affairs Australian Centre for International Agricultural Research
Australian Trade and Investment Commission
Department of Foreign Affairs and Trade
Tourism Australia
Health and Aged Care Australian Digital Health Agency
Australian Institute of Health and Welfare
Department of Health and Aged Care
Home Affairs Department of Home Affairs (Immigration and Border Protection) Industry, Science and Resources Australian Building Codes Board
Australian Nuclear Science and Technology Organisation
Commonwealth Scientific and Industrial Research Organisation
Department of Industry, Science and Resources
Geoscience Australia
IP Australia
Infrastructure, Transport, Regional Development, Communication and the Arts Australian Transport Safety Bureau Parliamentary Departments (not a portfolio) Department of Parliamentary Services Social Services Australian Institute of Family Studies
National Disability Insurance Agency
Treasury Australian Prudential Regulation Authority
Australian Securities and Investments Commission
Australian Charities and Not-for-profits Commission
Australian Taxation Office
Department of the Treasury
Productivity Commission
C.2 Issue register participation
Eleven agencies contributed to the evaluation through the Copilot issues register.
Agency Number of contributions[1] Australian Building Codes Board < 5 Australian Centre for International Agricultural Research < 5 Australian Digital Health Agency 77 Australian Prudential Regulation Authority < 5 Bureau of Meteorology 20 Comcare < 5 Commonwealth Scientific and Industrial Research Organisation < 5 Department of Agriculture, Fisheries and Forestry 35 Department of Industry, Science and Resources 56 Digital Transformation Agency < 5 Regional Investment Corporation < 5 Note: A asterisk denotes less than 5 participants
C.3 DTA outreach interview participation
Twenty-four agencies contributed to the evaluation via DTA outreach interviews.
- AgriFutures
- Australian Charities and Non-for-profits Commission (ACNC)
- Australian Criminal Intelligence Commission (ACIC)
- Australian Centre for International Agricultural Research (ACIAR)
- Australian Federal Police (AFP)
- Australian Institute of Family Studies (AFIS)
- Australian Institute of Marine Science
- Australian Prudential Regulation Agency (APRA)
- Australian Research Council (ARC)
- Australian Tax Office (ATO)
- Bureau of Meteorology (BOM)
- Comcare
- Commonwealth Ombudsman
- Commonwealth Scientific and Industrial Research Organisation (CSIRO)
- Department of Agriculture, Fisheries and Forestry (DAFF)
- Department of Home Affairs
- Department of Industry, Science and Resources (DISR)
- Department of Education, Employment and Workplace Relations (DEEWR)
- Geoscience Australia
- Intellectual Property (IP) Australia
- National Disability Insurance Agency
- Parliament of Australia (APH)
- Regional Investment Corporation (RIC)
- Tertiary Education Quality and Standards Agency (TEQSA)
C.4 Nous focus group participation
Sixteen agencies contributed to the evaluation through Nous-facilitated focus groups.
Agency Number of participants Australian Charities and Not-for-profits Commission < 5 Australian Digital Health Agency 5 Australian Institute of Family Studies < 5 Australian Institute of Health and Welfare < 5 Australian Space Agency < 5 Australian Transport Safety Bureau < 5 Bureau of Meteorology 7 Commonwealth Superannuation Corporation 6 Department of Agriculture, Fisheries and Forestry < 5 Department of Health and Aged Care < 5 Department of Industry, Science and Resources 13 Department of Parliamentary Services < 5 Digital Transformation Agency < 5 Infrastructure Australia < 5 IP Australia 6 National Disability Insurance Agency < 5 C.5 Nous interview participation
Eight agencies contributed to the evaluation via Nous-facilitated interviews.
Agency Number of participants Commonwealth Scientific and Industrial Research Organisation < 5 Department of Customer Service (NSW) < 5 Department of Industry, Science and Resources < 5 Digital Transformation Agency < 5 Office for Women < 5 Office of the Australian Information Commissioner < 5 Productivity Commission < 5 Services Australia < 5 C.6 Pre-use survey participation
Thirty-six agencies contributed to the evaluation via the pre-use survey.
Agency Number of participants AgriFutures Australia 13 Australian Building Codes Board 7 Australian Centre for International Agricultural Research 17 Australian Charities and Not-for-profits Commission 23 Australian Criminal Intelligence Commission 12 Australian Digital Health Agency < 5 Australian Financial Security Authority < 5 Australian Institute of Family Studies 16 Australian Renewable Energy Agency < 5 Australian Research Council 13 Australian Securities and Investments Commission 108 Australian Taxation Office 159 Australian Trade and Investment Commission < 5 Australian Transport Safety Bureau 6 Bureau of Meteorology 60 Comcare 62 Commonwealth Ombudsman 7 Commonwealth Scientific and Industrial Research Organisation < 5 Commonwealth Superannuation Corporation 100 Department of Agriculture, Fisheries and Forestry 41 Department of Climate Change, Energy, Environment and Water < 5 Department of Education 46 Department of Employment and Workplace Relations 57 Department of Finance < 5 Department of Health and Aged Care 125 Department of Home Affairs 93 Department of Industry, Science and Resources 163 Department of Parliamentary Services 53 Digital Transformation Agency 116 Fair Work Commission 8 Fair Work Ombudsman 7 Geoscience Australia 44 Grains Research and Development Corporation 14 IP Australia 37 Productivity Commission 10 Regional Investment Corporation < 5 Tertiary Education Quality and Standards Agency 11 C.7 Post-use survey participation
Twenty agencies contributed to the evaluation via the post-use survey.
Agency Number of participants Australian Building Codes Board 5 Australian Charities and Not-for-profits Commission 8 Australian Criminal Intelligence Commission 21 Australian Digital Health Agency 48 Australian Institute of Family Studies 8 Australian Nuclear Science and Technology Organisation < 5 Australian Radiation Protection and Nuclear Safety Agency 17 Australian Securities and Investments Commission 98 Australian Taxation Office 90 Australian Transport Safety Bureau < 5 Bureau of Meteorology < 5 Department of Education < 5 Department of Employment and Workplace Relations < 5 Department of Finance 107 Department of Health and Aged Care 60 Department of Home Affairs 10 Department of Industry, Science and Resources 142 Digital Transformation Agency 55 Infrastructure Australia 11 IP Australia 50 National Disability Insurance Agency 87 -
Appendix D
-
Appendix D: Survey participation by APS classification and job family
D.1 Survey participation by APS classification
Percentage of all APS employees (%) Percentage of pre-use survey respondents (%) Percentage of post-use survey respondents (%) SES 1.9 4.7 5.3 EL 2 9.0 20.0 20.2 EL 1 20.8 36.9 34.0 APS 6 23.4 23.4 22.3 APS 5 14.7 8.5 9.6 APS 3-4 26.0 6.0 7.4 APS 1-2 4.2 10.5 1.1 D.2 Survey participation by APS job family
Percentage of all APS employees (%) Percentage of pre-use survey respondents (%) Percentage of post-use survey respondents (%) Accounting and Finance 5.1 5.3 3.5 Administration 11.4 9.0 8.9 Communication and Marketing 2.5 4.9 5.8 Compliance and Regulation 10.3 6.6 6.5 Data and Research 3.7 9.9 8.3 Engineering and Technical 1.8 1.3 1.5 Human Resources 3.9 5.3 5.0 ICT and Digital Solutions 5.0 19.6 22.3 Information and Knowledge Management 1.1 2.5 1.6 Intelligence 2.4 0.9 2.1 Legal and Parliamentary 2.6 4.1 3.5 Monitoring and Audit 1.5 1.1 1.0 Policy 7.9 13.7 14.4 Portfolio, Program and Project Management 8.3 8.6 7.5 Science and Health 4.2 1.6 2.1 Senior Executive 2.1 2.3 1.5 Service Delivery 25.5 2.7 4.0 Trades and Labour 0.7 0.9 - -
Appendix E
-
Appendix E: Agency reports and evaluations
Agency Report Australian Tax Office (ATO) Microsoft 365 Copilot trial Update Commonwealth Scientific and Industrial Research Organisation (CSIRO) Copilot for Microsoft 365; Data and Insights Department of Home Affairs (Home Affairs) Copilot Hackathon Department of Industry, Science and Resources (DISR) DISR Internal Mid-trial Survey Insights -
Appendix F
Connect with the digital community
Share, build or learn digital experience and skills with training and events, and collaborate with peers across government.