Search

Lucy Poole speaking:

Good morning, everyone, and thank you for joining us today. My name is Lucy Poole. I'm the head of the Division for Strategy, Planning and Performance here at the Digital Transformation Agency. I'd like to advise our attendees today that this session will be recorded. However, as it's a webinar, only the presenters who present will be in the recording.

Before I begin, I would like to acknowledge the Traditional Custodians of the land on which we are all meeting on today, the various lands ... for me that is the Ngunnawal people. And I would like to pay my respects to any First Nations people that may be joining us today.

So firstly, it's wonderful to see such interest in our briefing today. I think, numbers are heading up towards 500. So that's an excellent turn out for us.

As you can see from our agenda, we've got a lot to cover off. And in a moment, I'll explain the context for the trial within our wider work on AI in government. I'll then pass on to Lauren Mills, who will give an overview of the evaluation approach, findings and, most importantly, the recommendations and consequently, what's next for government.

Before we go any further, I do need to let you know about a probity matter. So on the 4th of October, the DTA released an RFT on AusTender for the Cloud Marketplace refresh. We're also also working on the tender for the Digital Marketplace Panel too. To maintain fairness, we can't speak to not answer questions about these processes today, even if they seem simple or confirm public information. If you have direct questions about them, please reach out to the respective contact offices. Those details are shown on your screen.

I would also like to clarify from the outset that the Digital Transformation Agency is not responsible for how government departments or agencies choose to procure or adopt generative AI tools, including Copilot for Microsoft, 365. Now, with that out of the way, let's go to setting the scene for today. First of all, I would like to give thanks to everybody who's here and who provided questions ahead of today's session.

Your questions have helped us to frame up the conversation that we'll have with you today. And, with any luck, we'll cover off the majority of those questions. Based on what you told us, we're going to establish where the trial fits in the larger context of AI in government, what the recommendations mean for APS agencies and vendors, and where we go from here.

There are lots of specific questions that go deep into ways to use the product, or very specific technical and security implications. We could talk for hours on these aspects, but sadly we don't have the time. So this morning's session will include some resources that go some way to explaining the wider picture of AI in government. It will also cover all of the technical and security information about Microsoft 365 Copilot. For those who haven't read it, I highly encourage reading the full evaluation report. It answers almost all the questions that won't be directly addressed today. We've also enabled the Q&A functions in Teams. Our teams will try to answer as many of your questions as we can. Please do ask for follow-ups throughout the session today. We'll capture everything you ask and look to publish more information after the session to fill any gaps that we don't meet today.

So let's move into the wider context for AI in government. For those of you that aren't familiar with the work of the Digital Transformation Agency, we are government's adviser for the development, delivery and monitoring of whole-of-government strategies, policy, standards for digital asset investments and procurement. This includes setting the overall direction for how the Australian Government explores and procures and adopts new technology, including generative AI. We are responsible for what we call the whole of economy policy work on AI, those that impact every business or person across Australia. But we do ensure that our work aligns to this broader picture, because government should naturally be the exemplar of safe, responsible use of AI that fulfills Australia's AI ethics principles.

We work very closely with the Department of Industry, Science and Resources, who are responsible for this whole of economy piece. Together, we co-led the AI and Government Taskforce through to the end of June this year. The work of that taskforce has directly informed the work of the DTA, including the Whole-of-government Policy for the Responsible Use of AI, accompanying standards and guidance to help agencies to fulfill the policy's requirements, training and supporting guidance for APS staff, the end users of generative AI. These instruments are available and where relevant applicable right now. We also have an ongoing slate of work which will see results through the new year. These include developing AI technical standards for use by government, which will be openly available for use by other other governments, organisations and industry.

We are also working on piloting the Australian government's own AI assurance framework for positively managing the risks associated with different AI use cases and continuing our work on progressively updating the AI policy to keep up with both changes in technology and also the expectations of the APS and the wider community. It's within this context that we undertook the trial of the Microsoft 365 Copilot as an example of a generative AI tool, so that we can understand what impact this technology might begin to have on the way that public servants go about their work.

That was the intent of the trial and its evaluation: its impact on how people work. At the time of the trial, Microsoft 365 Copilot was the most appropriate tool to undertake an evaluation of general generative AI capabilities in day-to-day technology suites of the APS. Its integration with Office products familiar to our APS staff within existing whole-of-government contracting arrangements allowed us to undertake the evaluation in a timeline that worked for our needs and in a way that would ensure a relatively consistent implementation and user experience across government.

Now I'm starting to get into the weeds of the trial itself and how the evaluation worked. So at this point, I'll hand over to Lauren Mills. Thank you.

Lauren Mills speaking: 

Thanks, Lucy, and good morning, everyone. I'm Lauren Mills. I lead the Strategy and Prioritisation Branch here at the DTA. I'm going to start with a brief explanation of our evaluation approach, noting the full report is available. And it does go into a lot more detail for those that are interested. I think the team are going to popa link in the chat here, so if you haven't had the opportunity to read it, we just like to follow along today. You can grab that now. So as Lucy mentioned, the tool we selected for the trial was Microsoft 365 Copilot, which is ubiquitous about throughout the M365 suite of products. And so, given this, we thought that it wasn't the best approach to set a strict set of use cases for agencies to use.

So, for the avoidance of all doubt, just to be clear, we specifically looked at the Microsoft 365 Copilot and not other Copilot offerings. So, defining use cases was going to be counter-intuitive to the experimental nature of the trial. And we wanted to give flexibility to agencies with their specific operating environments and their own requirements to effectively choose their own adventure and see what they could find with the product. This was really important because we had over 7700 licenses purchased across 60 Australian Government entities participating in the trial. But we wanted to set up front with what do we need to find out through the course of the trial? So we worked closely with the AI and Government Taskforce, and we identified four key outcome areas that we wanted to explore.

So that was really around employee-related outcomes. So what about staff sentiment in the use of Copilot as an example of generative AI, including staff satisfaction, opportunities for innovation, confidence in the use of Copilot and how easily we could integrate it into our existing workflows? Of course, productivity was a key area we wanted to explore through the trial, both in terms of efficiency but also in quality, and whether there was opportunity for process improvements.

We also wanted to look at the adoption of AI more broadly, and to what extent Copilot could be implemented in a safe and responsible way across government, how it could pose benefits and challenges in the short and longer term, and also what barriers to innovation do we have that might require changing the way that we deliver our services to embrace the opportunities that these new technologies.

We also wanted to understand any unintended consequences, both benefits and challenges of implementing Copilot and the implications for broader adoption across the APS. So, the report that you've seen published includes the post-evaluation findings but I also wanted to share some of the insights we've learned throughout the trial. As I mentioned, one of the biggest challenges we had was the sheer breadth of agencies who had signed on to participate in the trial.

So every agency had a different level of maturity in their use of AI, different operating environments and, of course, different risk appetites. However, although all very different across them, these agencies, there were a lot of common themes that came through, and I'm just going to talk through some of those now. Initially, security was a key focus area, and agencies had various levels of reliance on the Australian Cyber Security Centre's infosec registered assessors program or IRAP assessments.

So while the IRAP actually assists agencies in their security assessments, it is of course not mandatory across the APS. However, in many cases, smaller agencies, particularly smaller agencies, have mandated it as part of their establishing their ability to authority to operate. So it was absolutely critical that we got that IRAP sorted straight, straight away. Another key area was data governance. While not a new risk for us, the nature of the tool and the ease with which Copilot could highlight the access to all the documents and files that individual staff members had access to.

So that was something we needed to understand exactly from the start. Some agencies took advantage of it and actually used Copilot to undertake audits of the information systems before fully rolling out the product. Others were satisfied that they had the right risks management processes in place to identify, and they put in processes to remediate anything found through the course of the trial.

The other key area was privacy. And similarly, and in some cases directly related to data governance, there were various levels of risk appetite in relation to privacy, and agencies who held customer data actually applied a higher level of caution, which makes sense. However, one of the biggest challenges was actually understanding what were the specific privacy considerations to generative AI. What made it different to other technologies?

We had established a program board to govern the trial, and underneath this program will be established a privacy working group to unpack some of these privacy considerations in more detail. As part of this group, we actually set up a cross-APS Privacy Impact Assessment, which was coordinated by the Department of Home Affairs to establish a base set of common assumptions and use cases.

Well, obviously, agencies are responsible for conducting their own assessments based on their own operating environments, they could use this joint PIA to reduce the duplication of effort as well as to reduce costs. And it was a really great example of how the APS can collaborate to efficiently solve problems and share our learnings. Finally, another one was recordkeeping. Through the central issues register that we established for the trial, there was a common theme around what constitutes a record under the Archives Act, specifically for things like meeting recordings, transcripts and, of course, first also documents using Copilot.

To unpack this further, we established a second working group under the Trials Program Board, which worked in consultation with the National Archives of Australia to establish a whole-of-government advice on how these records should be treated. This work remains ongoing but, at a high level, Copilot and other generative AI assistants could be viewed as simply another tool that staff may use to conduct their work and should have record retention periods that reflect that.

This is not a catchall, and there will be additional nuanced advice depending on this scenario or use case. And so as I said this, this work is continuing and we're hoping to get some whole of government advice out. In terms of what we saw around the success of the trial, through the course (of it), we saw that the agencies had the most success in the adoption and the use of this product were those that had already thought about their specific environments and the application of generative AI within their organisation, which makes sense.

And I think, the other key finding, which is around (something that is) quite common in change projects: those who had champions, particularly those who had strong executive sponsors promoting the benefits saw the highest adoption and were able to conduct robust internal evaluations.

So in terms of the post-trial evaluation findings, as I've mentioned, the reports are very detailed and will provide a really good source of information for all of you. So I'm going to cover off some of the findings just at a high level only today. As I mentioned earlier, there were four key areas that we looked at in terms of evaluation.

And so in terms of employee-related outcomes, we saw that most trial participants were positive about Copilot and wish to continue using it. So 86% of participants said they wanted to keep using the product. Interestingly, senior executive service staff, about 93%, and corporate roles, about 81%, had the highest positive sentiment towards copilot. However, despite the positive sentiment, use of Copilot was moderate.

Our analysis was conducted across both job families and those different levels across the APS, and moderate usage was consistent across these classifications and job families, but those specific use cases varied. For example, a high proportion of SES and EL2 staff used the meeting summarisation pages compared to other APS classifications, which makes sense. Microsoft Teams and Word were the most frequently used and met participants' needs.

However, there was considered to be very poor Excel functionality, and the access issues in Outlook did hamper use in that. In that product. As expected, content summarisation and rewriting were the most used Copilot functions, but it was clear that other generative AI tools might be more effective at meeting users' needs in terms of things like writing code, generating images or searching research databases.

It's clear that tailored training and propagation of high-value use cases could improve adoption. So we saw that training significantly enhanced confidence in the use of Copilot and was most effective when it was tailored to an agency-specific context. It's also important that we identify specific use cases for Copilot, which will help, as I said, help that adoption and promote that use of the product.

In productivity, most trial participants believed Copilot improved the speed and quality of their work, so improvements in both efficiency and quality were seen, with perceived time savings of around an hour a day for some people, including, for some of those tasks, such as summarisation, preparing that first draft of the document, or information services. And I think, really great to see that 40% of survey respondents reported reallocating their time for some important activities, such as mentoring and culture building, strategic planning, engaging with stakeholders, and product enhancement.

However, Copilot's inaccuracy did reduce the scale of these productivity benefits, so the gains in quality were more subdued relative to those efficiency gains. And the potential unpredictability and lack of contextual knowledge in terms of those outputs from Copilot required time spent on output verification, which kind of negated some of the efficiency savings.

In terms of adoption, there is, as I said, a need for agencies to engage in planning activities about how they bring onboard generative AI tools and making sure that those governance structures and processes appropriately reflect their risk appetites. Many of the insights under this outcome reflect what we found during the trial. But some of the key barriers were the integration challenges with non-Microsoft 365 applications. However, it should be noted that these integrations were actually out of scope for the trial.

Prompt engineering, identifying relevant use cases and understanding the information requirements of Copilot across the Microsoft Office products with significant capability barriers and, of course, that planning, and to reflect the rolling released nature gen AI tools alongside AI-relevant governance structures, is important.

And finally, to some of the unintended outcomes, some both benefits and concerns that will need to be actively monitored throughout the adoption of AI. So in terms of benefits, a really interesting outcome was that gen AI could improve inclusivity and accessibility in the workplace, particularly for those who are neurodiverse, with a disability or from a culturally and linguistically diverse background, and that the adoption of Copilot and gen AI more broadly could actually help the APS attract and retain employees.

However, there were some concerns, particularly around the potential impact of gen AI on APS jobs and skills needs in the future. Also, as we've seen more broadly, that the outputs might be biased towards our Western norms and may not appropriately use cultural data and information, such as misusing First Nations images and misspelling First Nations words. There was concerns that the use of Gen AI might lead to a loss of skill in summarisation in writing, or conversely that a lack of adoption of gen AI may result in the false assumption that people who use it might be more productive than those who don't.

Participants also expressed concerns relating to vendor lock-in. However, the report found the realised benefits were limited to specific features and use cases, as we've discussed. Finally, participants were also concerned about the APS's increased impact on the environment resulting from gen AI use. So in terms of the recommendations, the overarching findings revealed several considerations for the APS in the context of future adoption of AI. And we put together eight recommendations in total across three focus areas. So we need to ensure we do detailed and adaptive implementation. So in terms of product selection, agencies should consider which gen AI solution is most appropriate for their overall operating environment and their specific use cases, particularly for these AI assistant tools. In system configuration, we must configure our information systems, permissions and processes to safely accommodate gen AI products.

Specialised training is essential, reflecting agency-specific use cases and developing broader gen AI capabilities, including prompt training. As I discussed, change management is key. Effective change management should support the integration of gen AI and potentially identifying generative AI champions to highlight the benefits and encourage adoption. We need to develop clear guidance on using gen AI, including when consent and disclaimers are needed, such as in meeting recordings, and a clear articulation of accountabilities.

We need to encourage greater adoption through analysing our workflows across various job families and classifications to identify the use cases that can improve adoption.

We need to continue to share use cases. We've seen great collaboration and bringing together of knowledge across the APS around the use of this emerging technology. We need to continue that, and look where we can share those appropriate follow government forums to facilitate the adoption of gen AI.

And finally, we've talked a lot about some of these impacts, and we need to proactively monitor the impacts of generative AI, including its effects on the workforce, to manage current and emerging risks effectively. So what's next? Many of the questions we've received from you were future focused, which is great to see. The findings from the trial will directly inform the next iteration of the Policy for the Responsible Use of AI in Government, as well as the AI assurance framework, which Lucy mentioned we are piloting right now.

In addition to this, will continue to explore the work of privacy and recordkeeping under those working groups, and these will continue to remain in place post the conclusion of the trial. And we're working closely with the National Archives Australia, as well as the Office of the Australian Information Commission, to progress that work.

In terms of gen AI adoption across government, the decision to adopt generative AI remains the responsibility of each agency, as Lucy made clear upfront. We do play a role here at the DTA in supporting that decision-making through our policies and frameworks and access to vendors through the Digital Marketplace. Specific to Copilot, we do understand there may be an uptake across the service, and so we are currently finalising some technical readiness documentation to support agencies who do choose to implement Copilot.

This suite of documents aims to support that safe and responsible implementation of the product within the Australian Government context. And we're also working closely with the Australian Cyber Security Centre to complete that work. So that brings us to the end of that section and moving into the Q&A part of today's agenda. As noted at the start of the brief, we have grouped our questions into key themes in order to cover as much as possible.

So we're going to start with the questions that were asked through the registration process and, where possible, some that were raised today. And my colleagues, I believe, are also responding to questions through the Q&A function.

Lucy Poole speaking:

Thank you so much, Lauren. We have had a fairly significant disruption to our team's service our end, which means we're not able to access the questions coming through, nor even make some workarounds of the more manual type.

So what we're going to do is we're going to ... we're going to call it a day here. And what we will do is endeavor to, as I mentioned earlier, that the presentation today will be shared online. So you'll have full access to that, but there will be more information that the Digital Transformation Agency looks to push out based on the questions that we know that have come through that may not have been addressed throughout the session.

And I do appreciate the thumbs up and the clapping that's coming through. You know, whenever you're doing a live session, you've got to expect these things, I guess. So, I'd like to just give thanks to both the DTA team in the background who are, you know, frantically trying to fix the problem, but also those that were that were involved directly in the pilot.

So, thank you to Lauren Mills and, most importantly, thank you all for attending today. Your interest is greatly appreciated and we hope that the information that we've covered has been helpful for you. So, thank you. Enjoy the rest of your Friday. And have a fabulous weekend. Thank you.