Home > The Values and relations with the Public > Measuring the effectiveness of service delivery

MEASURING THE EFFECTIVENESS OF SERVICE DELIVERY

Both agency and employee survey results indicate that APS agencies are recognising the importance of obtaining their clients’ perspectives on service delivery.

To examine both how agencies were measuring the effectiveness of service delivery to their clients and the level of client satisfaction across a broad selection of the APS, the PS Commissioner wrote to the agency heads of eight large agencies with important service delivery functions and the agency head of a small agency requesting information on agency client surveys conducted in recent years, which examined satisfaction with agency service delivery. Information requested included survey methodologies, survey instruments, and time series data gathered over a five-year period, to allow the Commission to examine trends in client satisfaction.

Seven large agencies provided information. They were ATO, Centrelink, CSA, Customs, DEWR, DITR and DVA. The AWM was also included as an example of a small agency that regularly surveys its clients, bringing the total sample size to eight. All agencies provided a range of client survey information and data. The majority of agencies were able to provide four years of time series (trend) data and two agencies were able to provide trend data back to 1997—98 on client satisfaction with service delivery.

FOCUS ON SURVEYING AGENCIES’ CLIENTS

There is a wide range of surveying being undertaken by agencies in the sample. Agencies are surveying customers, clients, stakeholders, providers and the general public in order to gain a better understanding of how their business operations, programs, policy and staff knowledge and behaviours are perceived by their client base. Surveying is often only one facet of a range of market research undertaken by agencies.

All agencies in the sample have used client surveys to measure satisfaction with service delivery (with most using more than one method of obtaining client feedback) and there was evidence of an increasing use of surveys in recent years. The level of complexity and sophistication of surveying in agencies varies and appears to be dependent on a range of factors, including how well agency market research is integrated with ongoing business processes and practices, agencies’ commitment to measuring client satisfaction and the use of results as part of measuring agency performance.

The frequency with which surveys are conducted in the sample varies from agency to agency. Some surveys are conducted on an ad hoc basis, some are years apart, whereas others are conducted annually and/or biannually or even on an almost continuous basis, allowing agencies to gather trend data on the effectiveness of their service delivery. Of the eight in the sample, three have between five and seven years of continuous trend data, three have between three and four years of continuous trend data and two have intermittent data over two or more years.

A range of survey methodologies is used by the agencies to collect information from customers. These include the use of computer assisted telephone interviewing, face-to-face interviews, self-completion surveys, and the use of interactive voice response systems to identify and randomly select clients. This variety reflects both the needs of agencies and their particular clients, and the extent to which surveying is embedded within their business systems.

The majority of agencies in the sample (six) use contracted organisations which specialise in customer research to undertake the agency’s market research of their clients. Reasons for using contractors include expertise in the market research arena, resource availability, and perceived privacy and confidentiality issues for clients.

SAMPLING CLIENT POPULATIONS

The agencies in the sample used a variety of methods to identify an appropriate survey population. These included:

• random samples across whole of customer/client population
• stratified random samples of customer/client populations
• random selection by telephony system
• a census
• selection based on significant interaction with the agency or program.

Factors that contributed to agencies’ choice of sampling included size of population, ease of access to population, survey methodology, variety of groups within a population, most recent contact with the agency and the need to avoid survey fatigue of its clients.

Populations surveyed by the agencies fall into a range of groups and include:

• direct clients of agencies, for example, Centrelink customers, CSA clients, DVA’s veteran community, individuals and businesses dealing with the ATO and visitors to the AWM
• clients of external service providers, for example, jobseekers who use Centrelink and Job Network Members that provide services on behalf of DEWR
• providers of services on behalf of an agency, for example, DVA’s health service providers
• stakeholders, for example, DITR surveys industry peak bodies, professional associations, State government departments, academic researchers and business leaders
• the general public, for example, Centrelink and ATO.

Agencies generally survey those groups that deal most frequently with the agency. Although throughout this section the term ‘clients’ is used to describe those being surveyed, the above list highlights the variety of groups serviced by agencies, the different terms used to describe their clients and the types of relationships that are in place.

WHAT IS MEASURED?

All agencies in the sample measure a wide range of factors that contribute to the effectiveness of service delivery and their relationship with their clients. Some agencies also use their surveys as an opportunity to gain information about the effectiveness of program design and policy aspects of their business, though the focus of this study was the measures used for service delivery. Aspects of service delivery that have been measured by agencies can be grouped under the following:

• clients’ understanding of issues
• awareness and/or effectiveness of service charters
• overall satisfaction with service delivery
• accuracy and relevance of information provided by the agency
• personal characteristics of staff including staff attitudes/behaviours, for example, being courteous, friendly, helpful, polite, sincere, respectful and able to listen
• professional characteristics, such as being easy to contact, responsive, taking pride in the work, taking responsibility, and knowledge and understanding
• changes in satisfaction and/or expectations
• the ease with which clients are able to access government services.

USE OF SURVEY DATA BY AGENCIES

The extent to which agencies in the sample utilise survey data to improve service delivery appears to vary. Most agencies in the sample are reporting their survey data as trend data in their annual reports and are examining the findings of the survey within the agency. Other uses of data by agencies are to:

• establish and report key performance measures against business outcomes
• evaluate the effectiveness of specific segments and programs within agencies
• assist in the development of whole of agency or other strategies
• measure staff productivity with pay linked to the survey findings
• provide feedback to teams as a means of improving service delivery
• identify specific needs important to different client populations, for example, to identify the need for training and education of clients and to improve the consistency of the client experience
• ensure clients receive appropriate services and understand their obligations, and to improve client access to agency services and programs.

OUTCOMES

Agencies in the sample provided the Commission with survey data for a key client survey that each agency had conducted over a period of time. In consultation with each agency, the Commission selected one key indicator of client satisfaction from the survey data where trend data was available that best represented overall client satisfaction with service delivery for that agency.1

Figure 4.1 plots client satisfaction with different agencies on a single graph to provide a visual representation of the range of satisfaction levels of clients of APS service delivery. Direct comparisons between agencies should be made with caution, as agencies conduct surveys in the context of their own business environment and use different methodologies. However, the broad tools of satisfaction and trends in client perceptions are of interest.

Results from client satisfaction surveys from the eight agencies included in the sample paint a positive picture when it comes to the overall satisfaction of clients with key APS organisations. Figure 4.1 shows that, generally, client satisfaction levels of agencies sampled are tracking at 70% or higher since 2000 with five of the agencies trending above 80% in the current year.

The AWM has the most satisfied clients. This is perhaps not unexpected as people choose to visit the Memorial, unlike the clients of the CSA, who are required to access CSA services. CSA clients generally advise the lowest satisfaction levels, although still well over 70%.

Fluctuations in the graph can be accounted for by a number of factors including seasonal changes, implementation of new initiatives and/or contracts, and/or new training of employees. Nevertheless for most agencies there has been a trend to increased levels of satisfaction in recent years.

Figure 4.1: Overall client satisfaction with APS service delivery

Source: Agency client survey data

Notes: Dotted lines indicate non-consecutive years of data collection, with the exception of DITR which represents benchmarked data collected over two years. (a) Centrelink’s measure is ‘overall quality of Centrelink’s people, services and information as either good or very good’. (b) DEWR’s measure is ‘overall jobseeker satisfaction with service provided by Job Network Members as satisfied or very satisfied’. (c) AWM’s measure is ‘overall would you say your visit met or exceeded your expectations?’. (d) CSA’s measure is its ‘Professionalism Index’, which measures client satisfaction with four key elements of CSA staff professionalism: personal characteristics, professional characteristics, treatment of clients, and outcome of the interaction. Results are based on an average result on a five-point index scale to record client satisfaction. For representation in this graph the CSA index has been converted to a percentage scale for comparison with other agencies. (e) Customs’ measure was the ‘overall satisfaction of clients with the service provided by Customs that scored 7 or more out of a score of 10’. (f) DVA’s measure of ‘clients satisfied or very satisfied with the service provided by DVA’. Results represented are the average results of surveys conducted within each financial year. (g) ATO’s measure is its ‘Satisfaction with Professionalism Average’, which measures ATO staff on their behaviour and ability including levels of empathy, provision of fair and just outcomes, communication skills and levels of accountability. Results are based on an average result on a five-point index scale to record client satisfaction. For representation in this graph the ATO index has been converted to a percentage scale for comparison with other agencies. (h) DITR’s (AusIndustry) measure is ‘overall customer satisfaction with service delivery’. DITR results are unweighted to allow comparison with earlier benchmark data, shown as 2000—01 but which covered 1999—00 to 2001—02 years. Weighted data for 2002—03 (89%) and 2003—04 (90%) is slightly higher than the unweighted data.

1 Data provided by the sample agencies may not cover all of an agency’s programs, and may represent only one aspect of the agency’s business.

CHALLENGES

While analysis of the agency survey data highlighted positive service delivery outcomes for agencies, a number of challenges in relation to measuring customer satisfaction with service delivery were also identified. Agencies are dealing with these challenges in a number of ways. The challenges identified are as follows.

• Obtaining a satisfactory response rate to agency client surveys and ensuring that they represent client populations. Response rates to agency surveys considered as part of this analysis varied from as low as 10% to a high of 86% at Customs. Some agencies have had to consider as part of their survey methodology the issue of survey fatigue. Others have worked on refining their survey sampling, for example, by only surveying those clients who have had an interaction with the agency in the last three months, as opposed to the last 12 months.
• Balancing the need to collect trend data that is comparable over time and the ongoing need to continue to improve and/or change survey questions to better target aspects of agency service delivery.
• Finding the right balance in the frequency of the collection of data. A wide range of timeframes have been used by agencies to collect survey data: annually, biannually, biennially, in an ad hoc fashion. Some agencies are moving to a more regular timetable for collecting data and/or collect data on a monthly basis; others have reduced the frequency of surveys from six monthly to annually.
• Communicating results to clients. Agencies are collecting a wide range of data, but there is little evidence that feedback is being provided to client groups on the data collected.
• Ensuring agency employees are engaged in the process, especially where responses from surveys are intended for use as a tool to improve team performance and/or where employee cooperation is required in ensuring the methodology works.
• Making client surveys an integral part of the business, ensuring results are acted on to maximise agencies’ investment in survey research and to implement change.
• Distinguishing between service delivery and program outcomes, especially where clients see policy and service delivery as a complete package. This challenge was identified by a number of agencies in the sample, particularly those operating in a more contested environment.

The evidence from this examination of eight agencies is that the APS is continuing to pay closer attention to service delivery, and is applying increasingly sophisticated approaches to obtaining feedback and to disseminating that systematically to managers and operational staff. The experience of these agencies could be of benefit to other agencies.