Appendix 2: State of the service survey methodologies

Agency survey methodology

The scope of the agency survey was the 82 APS agencies, or semi-autonomous parts of agencies, employing at least 20 staff under the Act.

The 82 participating agencies were sent the online survey on 1 June 2005 for completion. Agencies were given six weeks to complete and submit their response.

As part of their survey return, agency heads were required to ‘sign off’ their agency’s response. All 82 agencies responded to the online agency survey. The results of the agency survey are one of the key sources of information which the Commission has relied on throughout the preparation of this report.

Employee survey methodology

The employee survey was designed to establish the views of APS employees on a range of issues, including work-life balance, the Values, the Code, job satisfaction, and learning and development. The results of the employee survey are one of the main sources of information on which the Commission has drawn during the preparation of this report.

The employee survey was also designed to complement the agency survey. The results of the employee survey were, in part, intended to act as a ‘reality check’ in analysing responses to the agency survey. To achieve this objective, similar questions were asked in both surveys on a range of topics. Additional questions, suitable for employees but not for agencies (such as those on job satisfaction and increasing individual productivity), were also included in the employee survey.

Scope and coverage

The scope of the employee survey was all APS employees (both ongoing and non- ongoing) in agencies with at least 100 APS employees. Employees in agencies that employed fewer than 100 APS employees were excluded on the basis that when tabulated their responses could possibly identify them.

The survey sample was drawn from APSED on 14 April 2005, at which time APSED indicated that the total number of APS employees was 131,125. The survey sample was selected from the total population of APS employees from agencies with at least 100 APS employees, which was 130,236. Appendix 1 provides information on agencies’ APS employee numbers as at 14 April 2005.

Stratification

A stratified random sample of 6602 APS employees was selected from APSED. The sample was stratified by:

• level (APS 1–6, EL and SES classification groups)
• agency size (small: 100–250 APS employees; medium: 251–1000 APS employees; and large: >1000 APS employees)
• agency (for the 21 large agencies, the four medium-sized portfolio departments and the Commission)
• location (ACT and non-ACT).

To enable sound statistical inferences to be made about all APS employees, individuals were randomly selected from each of the strata. Each individual within a stratum had an equal chance of selection.

The sampling rates varied between the strata to ensure that sufficient statistical accuracy would be achieved for survey estimates from APS employees with the key characteristics captured by the stratification variables (level, location, agency and agency size). To gain the same accuracy for estimates for a small population (such as the SES) a much higher sampling rate was required than for a larger population (such as APS 1–6 employees).

The accuracy requirements varied between the demographic variables listed above, and this also led to differing sampling rates for these demographic variables.

This stratification process has not introduced a bias in the population estimates because the responses are appropriately weighted to take these differing sample rates into account. See the section ‘Weighting and estimation’ below for further details.

Reporting of large agency results

The survey was designed to enable the 21 large agencies, the four medium-sized portfolio departments and the Commission to receive a copy of their own results from the employee survey for internal management purposes—subject to the results satisfying a statistical accuracy benchmark. For this to occur, these 26 agencies were included separately in the stratification process (see the section ‘Stratification’ above).

Where relevant, and to maintain consistency with previous years, the State of the Service report includes only agency level results of large agencies that met the minimum number of weighted responses (see the section ‘Measures of accuracy and error’ below). The medium-sized portfolio departments and the Commission are not included in any agency level analysis in the report.

Privacy, anonymity and confidentiality

Maintaining confidentiality throughout the entire employee survey process was a primary concern to the Commission.

Privacy arrangements for APSED preclude Commission staff, other than those in the APSED Team, the Group Manager of the Evaluation Group, and the Commission’s Executive, from accessing APSED data relating to individuals. This meant that the identity of those individuals selected in the sample from APSED was not available to the Commission’s State of the Service Team or any other non-APSED staff involved in the survey. A small number of ORIMA Research staff had access to the sample. All responses to the survey were anonymous so individuals could not be identified.

Each person invited to participate in the employee survey was provided with a unique password. This prevented multiple responses from individual respondents.

Survey design

The employee surveys conducted in previous years were used as the basis for this year’s survey. Some questions have been included on an annual basis, other questions have been cycled through on a two or three year basis, whilst others were included for the first time this year to address topical issues. To ensure the Commission maintains comparable time series data, any changes to questions repeated from previous years were kept to a minimum.

The draft employee survey was subjected to cognitive testing involving individuals at the APS 1–6 and EL classifications from the ATO, Centrelink, DEWR, Finance and Health. A further refined draft survey was then tested at a focus group session attended by individuals at the APS 1–6 and EL classifications from Centrelink, DEH, DEWR, DoTARS, FaCS, Finance and Health.

The majority of questions, 72 of 84, were asked of all respondents. Two questions were asked of SES employees only. Nine questions were asked of EL and SES employees only. One question was asked of APS 1–6 and EL employees but not of SES employees.

The questions asked of employees were not dependent on location (inside or outside the ACT) or agency size (small, medium or large).

The employee survey was delivered using two methods. The main delivery method was online via a password-protected Internet site. The majority of the sample was sent an email from ORIMA Research on behalf of the Commissioner inviting them to participate in the online survey.

A secondary, paper-based delivery method was developed and implemented for employees working in agencies that do not have access to an individual email account or do not have (or have only limited) access to the Internet. These employees received a letter from the Commissioner inviting them to participate in the survey, as well as a paper copy of the survey to complete and return to ORIMA Research.

The 6602 invitation emails and letters were sent out to the sample between 16 May and 1 June 2005.¹ Respondents were asked to complete the survey and submit or return it to ORIMA Research by Friday 10 June 2005.²

An adjustment was made to the final sample size to account for those out of scope of the survey (including repeatedly bounced emails, those ‘out of office’ for the entire survey period and those known to be no longer employed in the APS at the time of the survey).³ The final sample was reduced by 442 to 6160.

Weighting and estimation

The survey responses were re-weighted to reflect the characteristics of the underlying population of APS employees. This was done to ensure that the overall demographic characteristics (used for sample selection) of the survey results exactly matched the demographic characteristics of all APS employees. The re-weighting process was based on the four demographic characteristics used for selection of the sample, namely:

• level (APS 1–6, EL and SES classification groups)
• agency size (small: 100–250 APS employees; medium: 251–1000 APS employees; and large: >1000 APS employees)
• agency (for the 21 large agencies, the four medium-sized portfolio departments and the Commission)
• location (ACT and non-ACT).

There were, therefore, 168 different weights applied—level (3) multiplied by location (2) multiplied by agency size and agency (28). For this survey, the weights were calculated by dividing the populations of each stratum by the number of respondents to the survey in each stratum; for example, if there are 4000 ELs in medium-sized agencies in the ACT, and 200 responded, the weight assigned to each EL working in a medium-sized agency in the ACT is 20. If the data were not re-weighted, some strata could be overrepresented and others under-represented in the total survey results.

The weighting approach is based on that taken in previous years. The application of a uniform approach to sample selection and weighting will assist in the development of times series data.

The weighting approach adopted assumes that respondents respond in the same way as non-respondents for the characteristics of interest. The weighting method above assumes that the responding persons represent the non-responding persons.

In this survey, with a response rate of 59%, there would need to be a marked difference in the views of non-respondents from those of the respondents to alter or bias the overall results to any significant extent. For analysis presented in this report it was assumed that there was no significant bias between those who responded in the survey and those who did not respond.

The results are calculated under the assumption that responding persons answer in the same way as non-respondents. This should be considered when using the data to make inferences about the APS population.

Results have generally been presented rounded to the nearest whole percentage point (i.e. 38% not 37.7%). Due to this rounding, the percentage results for some questions may not add up to exactly 100%.

Measures of error and accuracy

Two types of error can occur in sample surveys: sampling error and non-sampling error. Sampling error arises because in a sample survey not all of the population are surveyed. Hence a measured sample statistic is not usually identical with the true population behaviour. Non-sampling errors cause bias in statistical results and can occur at any stage of a survey and can also occur with censuses (i.e. when every member of the target population is included). Sampling error can be estimated mathematically whereas estimating non-sampling error can be difficult. It is important to be aware of these errors, in particular non-sampling error, so that they can be either minimised or eliminated from the survey.

Non-sampling error

The survey received a response rate of 59%. This response rate excludes responses that were received but were insufficiently complete to provide input into the data generated. This response rate is very creditable for a voluntary survey.

Non-sampling errors can result from imperfections in reporting by respondents, errors made in recording and coding of responses, and errors made in processing the data. No quantifiable estimates are available on the effect of non-sampling errors. However, every effort was made to reduce the non-sampling errors to a minimum by careful survey design and efficient operating procedures. In particular, the online survey design minimised the possibility of errors made in recording and coding of responses, as the respondents themselves entered the data when responding to the survey.

In addition, identifiable errors made by respondents while completing the survey were removed from the results database; for example, responses made by APS 1–6 employees to an EL-only question have been removed to ensure the integrity of the data. Blank responses were generally coded to non-response categories. The exception to this practice arose where responses were needed for demographic items for weighting purposes. In instances where this occurred, survey responses were disregarded.

Sampling error

One measure of the sampling error of an estimate is the standard error. There are about 19 chances in 20 that a sample estimate will be within two standard errors of the true population value. This is known as the 95% Confidence Interval.

For instance, we are 95% confident that the estimate of the population that would be satisfied that their supervisor would support the use of flexible work practices is between 79.3% and 82.1% (an estimate of 80.7% and a confidence interval of +/-1.4% based on a standard error of 0.7%).

The following table illustrates the standard errors from the sample design associated with estimates from 12 key questions in the employee survey. Generally, the higher the sample size for a question, the lower the standard error; for example, questions following a ‘filter’ question are more likely to have a slightly higher standard error because the population size responding to that question is lower than for ‘non-filtered’ questions. The standard error for the performance pay question estimate, for example, is slightly higher than for many others because only employees with access to performance- related pay were asked the question, thereby reducing the sample size.

Results have not been reported for questions where the number of unweighted responses is fewer than 20. This approach has been adopted for two reasons: firstly, to eliminate the possible identification of individuals who responded to these questions; secondly, to remove less reliable results from the analysis. This approach has not affected reporting of results at the aggregate level; however, it has limited our ability to report on disaggregated data where the sample size is small—as is sometimes the case for questions following ‘filter’ questions.

It should also be noted that estimates relating to disaggregated data where the sample size is small will have a higher standard error because the population size responding to that question is lower than for aggregated data or disaggregated data where the sample size is large; for example, as can be seen from the following table illustrating the standard errors associated with estimates for disaggregated data, the standard error for Indigenous employees is higher than other standard errors because the population size responding to that question is small.

Interpretation of scales

Scales were included in any question that required a respondent to measure the strength or level of a theoretical construct. In its simplest form in the survey, a scale asked a respondent to rate the level of importance, satisfaction or effectiveness of various workplace variables on a five-point scale.

The scales used in the surveys were generally balanced—that is, they allowed the respondents to express one of the two extremes of view (e.g. satisfaction and dissatisfaction). These scales were also designed with a midpoint that allowed respondents to enter a ‘neutral’ response.

When interpreting scales it is important to realise that there is not an ordinal relationship between points in a scale. That is, the strength of opinion to shift a respondent from ‘neutral’ to ‘satisfied’ may be much smaller than the strength required to shift a respondent from ‘satisfied’ to ‘very satisfied’.

Summary indexes

Summary indexes have been used to assist analysis of results of a number of survey questions comprising several parts. The indexes operate to condense a multiple response question into a single index for comparative purposes; for example, in exploring respondents’ overall level of job satisfaction, a question comprising 15 factors was summarised into a single index using a point scoring system. In this way, analysis of the 15 job satisfaction factors can be supplemented by analysis at the summary level.

Coding of open-ended responses

The employee survey questionnaire provided specified response options for each question. It also included open-ended response options for some questions, which enabled respondents to provide a text response to a question. Open-ended options were commonly provided, for example, as part of a specified response question in the form of ‘other (please specify)’.

Some open-ended responses have been coded to assist analysis. Coding involved, for example, removing irrelevant and incidental comments from statistical outputs as well as counting relevant comments against the appropriate response option.

1 At the request of DEST, invitations to participate in the survey were sent to DEST employees on Wednesday 1 June 2005 to avoid overlap with DEST’s staff survey.

2 The return date was extended for all participants to 15 June 2005—this allowed DEST employees two weeks to complete the survey.

3 The sample was drawn in April 2005 and this was based on the most recent data provided by agencies, which was January 2005 for the majority (over 90%) of employees.

go to top of the page