Search

Set benchmarks or targets to evaluate service availability performance: 

• You might specify the minimum acceptable level of availability, the maximum tolerable level of planned and unplanned downtime, or the optimal range of load time for your service. See also Define clear objectives and Choose monitoring tools and methods.
• Establish a consistent schedule and procedure for tracking availability data and presenting the findings. 
• Examine the findings to pinpoint gaps, trends, patterns, or anomalies that highlight the service's performance and its impact on user outcomes.

Act on the findings and implement improvements or changes to the service design, delivery, or maintenance as needed. 

Actions may include:

• fixing bugs
• upgrading servers
• enhancing features
• communicating with users.

 See also Develop processes for continuous improvement.  

• DSS criterion 9. Monitor your service | digital.gov.au
• IT systems availability targets | Australian Taxation Office (ato.gov.au)
• Monitoring the status of your service - Service Manual - GOV.UK (www.gov.uk)
• Service Level Agreement | Western Australian Government (www.wa.gov.au)

• Understanding cohorts: Overlaying demographic data, such as location or socio-economic data, may help agencies to understand the outcomes of different cohorts and their interactions with digital services. If a cohort has a low success rate in completing transactions online, it may signal the need for digital service improvements.

• Compare performance to non-digital channels: While a user’s whole activity may involve multiple channels, including non-digital channels, in meeting Criterion 3, it is important that agencies measure the digital component(s) of the activity separately from the non-digital channels. For example, if an end-to-end service requires multiple digital transactions and an in-person assessment prior to the service being delivered, each digital component should be measured separately.

• Understand the touch points of a user’s digital journey: Capture data along the user’s digital journey. Map out the user flow and capture data from the start to the end of their journey, marked by the successful completion of their transaction. This data will help you recognise potential hurdles and drop-off points, to optimise how users complete their transactions online.
• Enhancing the overall experience: Many government services have both digital and non-digital channels, designed to work together. Consider assessing the user’s journey across different channels to help understand user behaviour and identify ways to enhance the digital experience, aiming to make the whole service smoother.

Begin by understanding what success looks like for your transactional digital service:

• Identify the user's objectives and goals through research, affirm their expectations, and make sure they match the agency's completion expectations. See also,Define clear objectives. 
• Identify the key tasks and/or transactions that users need to complete to achieve their goals. Break down the user journey into individual, manageable components, focusing on each critical interaction point. For example, if the service involves updating personal information, key tasks might include logging in, navigating to the profile section, editing details and saving changes.

Determine the measures of success for transactions: 

• Identify KPIs that indicate how well the service supports users to finish actions or tasks in the digital service. KPIs will depend on the nature of your service, and your user’s goals. For services involving applications or submissions, KPIs could include the number of applications started versus completed, the average time to submit an application, and the number of users who encounter errors and abandon the process. Refer also to Choose relevant metrics.
• Where feasible, DTA recommend measuring the number and percentage of digital tasks that are started and completed successfully versus abandoned.
• Additional metrics to monitor service success include:
  • the time spent by users on each touchpoint, task and the overall transaction
    • the sources of traffic that lead users to the digital service
    • errors, bugs, or technical issues encountered by users 
    • user behaviour patterns, such as clicks, scrolls, mouse movements, and keystrokes
    • user demographics, such as age, gender, location, and language spoken
    • Capturing a Customer Effort Score (CES) with a free-text feedback dialogue at points along the user journey. See also Criterion 4.
  • Once you have determined KPIs, choose tools to collect, store and analyse data. These tools should integrate with your systems and provide real-time insights and comprehensive reports. See also Choose monitoring tools and methods. 

Once the digital service is operational and monitoring tools are implemented:

• Regularly analyse data and report insights within your agency to build an organisational understanding of how helpful your digital service is and identify areas for improvement. 
• Act on the findings and implement improvements or changes to the service design, delivery, or maintenance, as needed. 
• Test the improvements and measure for impact by monitoring for changes in data. 
• Ensure that measures of success are embedded into an ongoing process of continuous improvement. See also Develop processes for continuous improvement).

• Measuring completion rate - Service Manual - GOV.UK (www.gov.uk)
• DSS criterion 9. Monitor your service | digital.gov.au
• Choosing digital analytics tools - Service Manual - GOV.UK (www.gov.uk)
• Australian Privacy Principles | OAIC

• Use a methodology that suits the digital service: Customer satisfaction is a widely implemented, industry-standard measure of digital service quality. There are many quantitative methods to measure user sentiment. When designing a methodology, implement effective mechanisms that suit the digital service. For example, users can quickly and easily choose ‘thumbs up’ or ‘thumbs down’ options, which results in high response rates. A feedback form with an open text field requires more effort from the user, but provides more specific insight as to why a customer has provided that rating.

• Design convenient feedback mechanisms: Design a feedback mechanism that is easy and accessible for users and encourages engagement. The higher the response rate, the closer the data will be to the true sentiment of users. Having accessible and prominent feedback channels across every webpage and digital service will result in more valuable insights to enhance the user experience.

• Listen to and understand user needs: By capturing and tracking customer satisfaction with the digital services, agencies can learn how users feel about the quality of their digital service(s) and the areas to be improved. This includes understanding user expectations and what they need from government digital services. Positive customer satisfaction indicates well-designed, accessible and inclusive digital services.

Choose an appropriate method and tool to monitor levels of user satisfaction:

• Consider the outcomes and goals of your digital service, and the needs and expectations of your users. See also: Define clear objectives.
• When choosing a tool, consider how to minimise the burden on your users, along with the frequency, timing, and sample size of your data collection.
• The DTA recommend a simple thumbs up/down sentiment tool on each page of the digital service that is followed up by a free text feedback dialogue. See also Choose monitoring tools and methods. 

Configure your selected tool as an easy way for users to provide feedback about the service: 

• Craft precise and straightforward questions or statements to gauge user satisfaction.
• Validate questions and the tool’s functionality by testing with a representative sample of users to ensure reliability and accuracy.
• Make sure customer feedback tools are available on each page and at the end of a digital transaction.

Regularly assess customer satisfaction with your digital service and make improvements:

• Implement a tool to measure the user satisfaction measurement in your digital service and monitor the results regularly. 
• Use data visualisation and reporting tools to communicate the findings and trends to your stakeholders.
• Evaluate the impact of your user satisfaction measurement on your digital service performance and use the insights to identify areas for continuous improvement or innovation. See also Develop processes for continuous improvement. 
• Test and iterate your digital service based on user feedback and satisfaction data.