Executive summary
In the few years since its public introduction, generative AI (artificial intelligence) has become available and accessible to millions of people. The growing availability and speed of uptake in publicly available tools, such as ChatGPT, meant the Australian Public Service (APS) had to respond quickly to allow its workforce to experiment with generative AI in a safe, responsible and integrated way. To facilitate this experimentation, an appropriate generative AI tool needed to be selected.
Microsoft 365 Copilot (formerly Copilot for Microsoft 365) was one of the solutions available to enable the APS to undertake safe and responsible generative AI experimentation. On 16 November 2023, the Australian Government announced a 6-month whole-of-government trial of Copilot.
This decision was predicated on how swiftly and seamlessly Copilot, as a capability nested within existing whole-of-government contracting arrangements with Microsoft, could be deployed for rapid APS experimentation purposes. Further, as Copilot is a supplementary product that integrates with the existing applications within the Microsoft 365 suite, it also allowed staff to experiment and learn within the context of applications that were already familiar to them.
The trial involved the distribution of over 5,765 Copilot licenses between January to June 2024. The trial was non-randomised, with agencies nominating staff to be allocated a license. Trial participants comprised a range of APS classifications, job families, experience levels with generative AI, and expectations of generative AI capabilities.
Further detail on the background of the evaluation can be found in Appendix A.
More broadly, this trial and evaluation tested the extent to which much of the wider promise of generative AI capabilities would translate into real-world adoption by workers. The results will help the government consider future opportunities and challenges related to adopting generative AI.
This was just the first trial of a generative AI tool within the Australian Government and the future brings exciting opportunities to understand what other tools are available to explore a broad landscape of use cases.
Overview of the evaluation
Nous Group (Nous) was engaged by the Digital Transformation Agency (DTA) to assist with the evaluation of the trial. The Australian Centre of Evaluation was consulted on methodology and approach to ensure best practise. The evaluation was guided by 4 objectives and key lines of enquiry (KLEs) outlined in the table below.
| Evaluation objective | Key lines of enquiry | |
|---|---|---|
| Employee related outcomes | Determine whether Copilot, as an example of generative AI, benefits APS productivity in terms of efficiency, output quality, process improvements and agency ability to deliver on priorities. | What are the perceived effects of Copilot on APS employees? |
| Productivity | Evaluate APS staff sentiment about the use of Copilot. | What are the perceived productivity benefits of Copilot? |
| Whole-of-government adoption of generative AI | Determine whether and to what extent Copilot, as an example of generative AI, can be implemented in a safe and responsible way across government. | What are the identified adoption challenges of Copilot, as an example of generative AI, in the APS in the short and long term? |
| Unintended outcomes | Identify and understand unintended benefits, consequences, or challenges of implementing Copilot as an example of generative AI and the implications on adoption of generative AI in the APS. | Are there any perceived unintended outcomes from the adoption of Copilot? Are there broader generative AI effects on the APS? |
The findings of the evaluation and the resulting implications are outlined at a high level in this executive summary. Further details are provided in the body of the report.
Glossary
| Term | Meaning |
|---|---|
| AI in Government Taskforce | Co-led by the DTA and the Department of Industry, Science and Resources (DISR), the AI in Government Taskforce aimed to deliver policies, standards, and guidance for the safe, ethical and responsible use of AI technologies by government. |
| Confidence interval | A confidence interval is a statistical concept used to estimate a population parameter based on sample data. It provides a range of values that likely contain the true population parameter with a certain level of confidence. |
| Generative AI | Generative AI is a branch of artificial intelligence focused on designing algorithms that generate novel outputs, such as text, images or sounds, based on learned patterns from data. |
| Hallucinations | Large Language Models (LLMs) such as Microsoft 365 Copilot are trained to predict patterns rather than understand facts, sometimes leading to it returning plausible sounding but inaccurate information, which is referred to as a ‘hallucination’. |
| Large Language Model (LLM) | Large language models are a category of foundation models trained on immense amounts of data making them capable of understanding and generating natural language and other types of content to perform a wide range of tasks. |
| Microsoft 365 | A cloud-based suite of productivity and collaboration tools offered by Microsoft, including Office applications, email and other services. |
| Microsoft Office | A suite of desktop productivity applications from Microsoft, including Word, Excel, PowerPoint and others. |
| Microsoft Graph | A Microsoft application programming interface (API) that provides access to data and intelligence across Microsoft 365 services, enabling developers to build apps that interact with organisational data. |
| Mixed methods | Combined use of both qualitative and quantitative research approaches to provide a more comprehensive understanding of the subject being evaluated. |
| Microsoft 365 Copilot | AI-enabled functionality embedded into the Microsoft 365 application suite. Formerly called Copilot for Microsoft 365. |
| P-value | A statistical measure that indicates the probability of observing a result as extreme as, or more extreme than, the one observed, assuming the null hypothesis is true. |
| Trial participant | An Australian Public Service staff member who participated in the whole-of-government Microsoft 365 Copilot trial, between January and July 2024. |
| T-test | A statistical test used to compare the means of 2 groups to determine if they are significantly different from each other, accounting for the variability in the data and sample size. |
Employee-related outcomes
Productivity
Whole-of-government adoption of generative AI
Unintended outcomes
Appendix A: Background
Appendix B: Methodology
Appendix C: Agency participation in the evaluation
C1. Overall participation
| Portfolio | Entity |
|---|---|
| Agriculture, Fisheries and Forestry | Department of Agriculture, Fisheries and Forestry Grains Research and Development Corporation Regional Investment Corporation Rural Industries Research and Development (trading as AgriFutures Australia) |
| Attorney-General’s | Australian Criminal Intelligence Commission Australian Federal Police Australian Financial Security Authority Office of the Commonwealth Ombudsman |
| Climate Change, Energy, the Environment and Water | Australian Institute of Marine Science Australian Renewable Energy Agency Department of Climate Change, Energy, Environment and Water Bureau of Meteorology |
| Education | Australian Research Council Department of Education Tertiary Education Quality and Standards Agency |
| Employment and Workplace Relations | Comcare Department of Employment and Workplace Relations Fair Work Commission |
| Finance | Commonwealth Superannuation Corporation Department of Finance Digital Transformation Agency |
| Foreign and Trade Affairs | Australian Centre for International Agricultural Research Australian Trade and Investment Commission Department of Foreign Affairs and Trade Tourism Australia |
| Health and Aged Care | Australian Digital Health Agency Australian Institute of Health and Welfare Department of Health and Aged Care |
| Home Affairs | Department of Home Affairs (Immigration and Border Protection) |
| Industry, Science and Resources | Australian Building Codes Board Australian Nuclear Science and Technology Organisation Commonwealth Scientific and Industrial Research Organisation Department of Industry, Science and Resources Geoscience Australia IP Australia |
| Infrastructure, Transport, Regional Development, Communication and the Arts | Australian Transport Safety Bureau |
| Parliamentary Departments (not a portfolio) | Department of Parliamentary Services |
| Social Services | Australian Institute of Family Studies National Disability Insurance Agency |
| Treasury | Australian Prudential Regulation Authority Australian Securities and Investments Commission Australian Charities and Not-for-profits Commission Australian Taxation Office Department of the Treasury Productivity Commission |
C.2 Issue register participation
Eleven agencies contributed to the evaluation through the Copilot issues register.
| Agency | Number of contributions[1] |
|---|---|
| Australian Building Codes Board | < 5 |
| Australian Centre for International Agricultural Research | < 5 |
| Australian Digital Health Agency | 77 |
| Australian Prudential Regulation Authority | < 5 |
| Bureau of Meteorology | 20 |
| Comcare | < 5 |
| Commonwealth Scientific and Industrial Research Organisation | < 5 |
| Department of Agriculture, Fisheries and Forestry | 35 |
| Department of Industry, Science and Resources | 56 |
| Digital Transformation Agency | < 5 |
| Regional Investment Corporation | < 5 |
Note: A asterisk denotes less than 5 participants
C.3 DTA outreach interview participation
Twenty-four agencies contributed to the evaluation via DTA outreach interviews.
- AgriFutures
- Australian Charities and Non-for-profits Commission (ACNC)
- Australian Criminal Intelligence Commission (ACIC)
- Australian Centre for International Agricultural Research (ACIAR)
- Australian Federal Police (AFP)
- Australian Institute of Family Studies (AFIS)
- Australian Institute of Marine Science
- Australian Prudential Regulation Agency (APRA)
- Australian Research Council (ARC)
- Australian Tax Office (ATO)
- Bureau of Meteorology (BOM)
- Comcare
- Commonwealth Ombudsman
- Commonwealth Scientific and Industrial Research Organisation (CSIRO)
- Department of Agriculture, Fisheries and Forestry (DAFF)
- Department of Home Affairs
- Department of Industry, Science and Resources (DISR)
- Department of Education, Employment and Workplace Relations (DEEWR)
- Geoscience Australia
- Intellectual Property (IP) Australia
- National Disability Insurance Agency
- Parliament of Australia (APH)
- Regional Investment Corporation (RIC)
- Tertiary Education Quality and Standards Agency (TEQSA)
C.4 Nous focus group participation
Sixteen agencies contributed to the evaluation through Nous-facilitated focus groups.
| Agency | Number of participants |
|---|---|
| Australian Charities and Not-for-profits Commission | < 5 |
| Australian Digital Health Agency | 5 |
| Australian Institute of Family Studies | < 5 |
| Australian Institute of Health and Welfare | < 5 |
| Australian Space Agency | < 5 |
| Australian Transport Safety Bureau | < 5 |
| Bureau of Meteorology | 7 |
| Commonwealth Superannuation Corporation | 6 |
| Department of Agriculture, Fisheries and Forestry | < 5 |
| Department of Health and Aged Care | < 5 |
| Department of Industry, Science and Resources | 13 |
| Department of Parliamentary Services | < 5 |
| Digital Transformation Agency | < 5 |
| Infrastructure Australia | < 5 |
| IP Australia | 6 |
| National Disability Insurance Agency | < 5 |
C.5 Nous interview participation
Eight agencies contributed to the evaluation via Nous-facilitated interviews.
| Agency | Number of participants |
|---|---|
| Commonwealth Scientific and Industrial Research Organisation | < 5 |
| Department of Customer Service (NSW) | < 5 |
| Department of Industry, Science and Resources | < 5 |
| Digital Transformation Agency | < 5 |
| Office for Women | < 5 |
| Office of the Australian Information Commissioner | < 5 |
| Productivity Commission | < 5 |
| Services Australia | < 5 |
C.6 Pre-use survey participation
Thirty-six agencies contributed to the evaluation via the pre-use survey.
| Agency | Number of participants |
| AgriFutures Australia | 13 |
| Australian Building Codes Board | 7 |
| Australian Centre for International Agricultural Research | 17 |
| Australian Charities and Not-for-profits Commission | 23 |
| Australian Criminal Intelligence Commission | 12 |
| Australian Digital Health Agency | < 5 |
| Australian Financial Security Authority | < 5 |
| Australian Institute of Family Studies | 16 |
| Australian Renewable Energy Agency | < 5 |
| Australian Research Council | 13 |
| Australian Securities and Investments Commission | 108 |
| Australian Taxation Office | 159 |
| Australian Trade and Investment Commission | < 5 |
| Australian Transport Safety Bureau | 6 |
| Bureau of Meteorology | 60 |
| Comcare | 62 |
| Commonwealth Ombudsman | 7 |
| Commonwealth Scientific and Industrial Research Organisation | < 5 |
| Commonwealth Superannuation Corporation | 100 |
| Department of Agriculture, Fisheries and Forestry | 41 |
| Department of Climate Change, Energy, Environment and Water | < 5 |
| Department of Education | 46 |
| Department of Employment and Workplace Relations | 57 |
| Department of Finance | < 5 |
| Department of Health and Aged Care | 125 |
| Department of Home Affairs | 93 |
| Department of Industry, Science and Resources | 163 |
| Department of Parliamentary Services | 53 |
| Digital Transformation Agency | 116 |
| Fair Work Commission | 8 |
| Fair Work Ombudsman | 7 |
| Geoscience Australia | 44 |
| Grains Research and Development Corporation | 14 |
| IP Australia | 37 |
| Productivity Commission | 10 |
| Regional Investment Corporation | < 5 |
| Tertiary Education Quality and Standards Agency | 11 |
C.7 Post-use survey participation
Twenty agencies contributed to the evaluation via the post-use survey.
| Agency | Number of participants |
| Australian Building Codes Board | 5 |
| Australian Charities and Not-for-profits Commission | 8 |
| Australian Criminal Intelligence Commission | 21 |
| Australian Digital Health Agency | 48 |
| Australian Institute of Family Studies | 8 |
| Australian Nuclear Science and Technology Organisation | < 5 |
| Australian Radiation Protection and Nuclear Safety Agency | 17 |
| Australian Securities and Investments Commission | 98 |
| Australian Taxation Office | 90 |
| Australian Transport Safety Bureau | < 5 |
| Bureau of Meteorology | < 5 |
| Department of Education | < 5 |
| Department of Employment and Workplace Relations | < 5 |
| Department of Finance | 107 |
| Department of Health and Aged Care | 60 |
| Department of Home Affairs | 10 |
| Department of Industry, Science and Resources | 142 |
| Digital Transformation Agency | 55 |
| Infrastructure Australia | 11 |
| IP Australia | 50 |
| National Disability Insurance Agency | 87 |
Appendix D: Survey participation by APS classification and job family
D.1 Survey participation by APS classification
| Percentage of all APS employees (%) | Percentage of pre-use survey respondents (%) | Percentage of post-use survey respondents (%) | |
| SES | 1.9 | 4.7 | 5.3 |
| EL 2 | 9.0 | 20.0 | 20.2 |
| EL 1 | 20.8 | 36.9 | 34.0 |
| APS 6 | 23.4 | 23.4 | 22.3 |
| APS 5 | 14.7 | 8.5 | 9.6 |
| APS 3-4 | 26.0 | 6.0 | 7.4 |
| APS 1-2 | 4.2 | 10.5 | 1.1 |
D.2 Survey participation by APS job family
| Percentage of all APS employees (%) | Percentage of pre-use survey respondents (%) | Percentage of post-use survey respondents (%) | |
| Accounting and Finance | 5.1 | 5.3 | 3.5 |
| Administration | 11.4 | 9.0 | 8.9 |
| Communication and Marketing | 2.5 | 4.9 | 5.8 |
| Compliance and Regulation | 10.3 | 6.6 | 6.5 |
| Data and Research | 3.7 | 9.9 | 8.3 |
| Engineering and Technical | 1.8 | 1.3 | 1.5 |
| Human Resources | 3.9 | 5.3 | 5.0 |
| ICT and Digital Solutions | 5.0 | 19.6 | 22.3 |
| Information and Knowledge Management | 1.1 | 2.5 | 1.6 |
| Intelligence | 2.4 | 0.9 | 2.1 |
| Legal and Parliamentary | 2.6 | 4.1 | 3.5 |
| Monitoring and Audit | 1.5 | 1.1 | 1.0 |
| Policy | 7.9 | 13.7 | 14.4 |
| Portfolio, Program and Project Management | 8.3 | 8.6 | 7.5 |
| Science and Health | 4.2 | 1.6 | 2.1 |
| Senior Executive | 2.1 | 2.3 | 1.5 |
| Service Delivery | 25.5 | 2.7 | 4.0 |
| Trades and Labour | 0.7 | 0.9 | - |