Appendix 2: Survey methodologies

Information in this section is adapted from the SOSR 2011–12. Further detail on these issues can be found in Appendix 3 of the SOSR 2011–12 (pp. 265–275).

Employee census methodology

The census population consisted of all APS employees (ongoing and non-ongoing), as recorded in the Australian Public Service Employment Database (APSED) on 4 May 2012^[27]. Micro-agencies are defined as those who had fewer than 100 employees at this time. The employee census was open between 8 May 2012 and 6 June 2012. A total of 87,214 employees took part, a response rate of 55%. Nine hundred respondents were from Micro-agencies, which is a response rate for these agencies of 71%.

To ensure confidentiality, each employee was provided with a unique password to prevent multiple responses from individuals. Only a small number of staff of the external survey provider, ORC International, had access to both individual names and their unique passwords. All responses provided to the Commission by ORC International were de-identified. Due to these precautions, APSC staff could not identify individual respondents to the census or identify those who had not taken part.

Agency survey methodology

A key source of data for the SOSR is the agency survey, which was completed by the 101 APS agencies, or semi-autonomous parts of agencies, employing at least 20 staff under the Public Service Act 1999. This excludes nine of the 32 Micro-agencies. Furthermore, the remaining Micro-agencies completed a shortened form of the survey consisting of sections A, B, C, D, G, H and N. Consequently, the agency survey has not been used in this report.

Analysis strategy

This draws on both quantitative and qualitative data.

Quantitative data

A total of 87,214 employees completed the employee census, 900 from the Micro-agencies, and the remaining 86,314 from larger agencies. With sample sizes this large, it is likely that even very small differences will be statistically significant. This is because as the sample sizes increase, the size of the difference required for a significance test to return a positive result decreases.

In order to avoid presenting misleading information by over-emphasising statistically significant but very small differences, an additional criterion has been considered: the size of the difference (the effect size).

Where a difference is statistically significant and greater than small in magnitude, it has been flagged in the text. Differences smaller than this are unlikely to be visible in the workplace or to form a sound evidence basis for pragmatic recommendations. These are generally not reported unless they reinforce other findings with larger effect sizes. Table 1 shows the statistical criteria used to determine small effect size. These guidelines have been published in Statistical Power Analysis for the Behavioural Sciences^[28] and are widely in the used social sciences.

Longitudinal analyses

Many of the items included in the 2012 employee census were also included in the 2011 Micro-agency Snapshot survey. Ordinarily this would mean that the 2012 results could be benchmarked against the 2011 results. However, the Micro-agency Snapshot survey was completed by staff from 13 agencies, while the employee census covered 32. Any change in the aggregated results may be due to the inclusion of 19 additional agencies. Longitudinal analyses are not performed because the two samples are too different and results are likely to be misleading.

Qualitative data

The employee census provided specified response options for most questions. Complementing these, several items were completely open-ended, asking the individual to provide a short, written response to a question or statement. Open-ended responses were used to complement information gained through quantitative methods. Not all respondents provided a response to an open-ended question and comments do not necessarily represent the views of all respondents. However, comments represent a rich data source.

Open-ended comment analysis was based on the grounded theory approach in which key concepts from the collected data were coded either manually or with text mining software (Leximancer). Comments were reported using themes and concepts rather than individual responses, except when comments were non-attributable and served to highlight especially salient concepts or themes.

[27] Over the survey period, employees who were recorded in agency HR systems, but had not yet been recorded in APSED were invited to take part in the census as they were identified.

[28] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, Psychology Press, New York, 1988.