Home page
> Indigenous affairs > Census report > Appendix: Aboriginal and Torres Strait Islander APS Employees Census Survey methodology
> Publications
‹ Previous page

Census report

related indigenous items Useful references

Appendix
Aboriginal and Torres Strait Islander APS Employees Census Survey methodology

The census survey was designed to gain a better understanding of the views of Aboriginal and Torres Strait Islander employees on a range of issues, including pathways to employment, work-life balance, job satisfaction, people management and learning and development. The results of the survey are the main source of information on which the Australian Public Service Commission (the Commission) has drawn during the preparation of this report.

Scope and coverage

The scope of the census survey was all APS employees (ongoing and non-ongoing) who were listed on the Australian Public Service Employment Database (APSED) as an Aboriginal or Torres Strait Islander.⁵⁶ Provisions were also made for Aboriginal and Torres Strait Islander APS employees who were not identified on APSED to register with the Commission to participate in the survey.

The survey population was drawn from APSED on 20 September 2005, at which time APSED indicated that the total number of employees identifying as Aboriginal and Torres Strait Islander was 2976.

Privacy, anonymity and confidentiality

Maintaining confidentiality throughout the entire survey process was a primary concern of the Commission.

Privacy arrangements for APSED preclude Commission employees, other than those in the APSED Team, the Group Manager of the Evaluation Group, and the Commission’s Executive, from accessing APSED data relating to individuals. This meant that the identity of the survey population was not available to the Commission’s Evaluation Team or any other non-APSED employees involved in the survey. A small number of ORIMA Research staff had access to the data. All responses to the survey were anonymous so that individuals could not be identified.

Each person invited to participate in the census survey was provided with a unique password. This prevented multiple responses from individual respondents.

Survey design

The census survey was designed based on a range of surveys previously conducted by the Commission, most notably the State of the Service employee surveys of 2004 and 2005.

The draft employee survey was subjected to cognitive testing involving individuals at the APS 1–6, Executive Level (EL) and Senior Executive Service (SES) classifications from the Australian Taxation Office (ATO), the Department of Employment and Workplace Relations (DEWR), the Department of the Environment and Heritage (DEH) and the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS). A further refined draft survey was then tested at two focus group sessions attended by individuals at the APS 1–6 and EL classifications from Centrelink, DEWR, DEH, the Department of Transport and Regional Services (DOTARS), the then Department of Family and Community Services (FaCS), the Department of Foreign Affairs and Trade (DFAT), Aboriginal Hostels Limited (AHL) and the Department of Health and Ageing (Health).

The survey was delivered using two methods. The main delivery method was online via a password-protected Internet site. The majority of Aboriginal and Torres Strait Islander APS employees were sent an email from ORIMA Research on behalf of the Commission inviting them to participate in the online survey.

A secondary, paper-based delivery method was developed and implemented for employees working in agen-cies that do not have access to an individual email account or do not have (or have only limited) access to the Internet. These employees received a letter from the Commission inviting them to participate in the survey, as well as a paper copy of the survey to complete and return to ORIMA Research.

2999 invitation emails and letters were sent out between 10 and 15 November 2005. Respondents were asked to complete the survey and submit or return it to ORIMA Research by Friday 2 December 2005.⁵⁷

An adjustment was made to the final survey population size to account for those out of scope of the survey, including:

a small number of employees who could not be contacted (covering repeatedly bounced emails, returned hard-copy surveys where the address was unknown and those ‘out of office’ for the entire survey period)
those known to be no longer employed in the APS at the time of the survey
those who were incorrectly identified on APSED as being Aboriginal or Torres Strait Islander.

Adjustments were also made to account for those Aboriginal and Torres Strait Islander APS employees who were not identified on APSED, but who had registered with the Commission to participate in the survey.⁵⁸ The final survey population was reduced by 371 to 2628.

Weighting

The survey responses were re-weighted to correct for different response rates between strata. This was done to ensure that the aggregate results represent the underlying demographic profile of Aboriginal and Torres Strait Islander APS employees. The re-weighting process was based on the following demographic variables:

level (APS 1–2/trainee/cadet, APS 3–4/graduate, APS 5–6, EL 1, EL 2, SES)
agency (for the 49 agencies where respondents were employed)
location (ACT and non-ACT).

There were, therefore, 588 different weights applied—level (6) multiplied by agency (49) multiplied by location (2). For this survey, the weights were calculated by dividing the overall response rate by the response rate for each stratum; for example, if there are 20 APS 5–6 Aboriginal and Torres Strait Islander APS employees in the Commission in the ACT, and the response rate for this group was 80% (with the overall response rate being 59%), the weight assigned to each APS 5–6 respondent working in the Commission in the ACT is 0.74. If the data were not re-weighted, some strata could be over-represented and others under-represented in the total survey results.

Results have generally been presented rounded to the nearest whole percentage point (i.e. 38% not 37.7%). Due to this rounding, the percentage results for some questions may not add up to exactly 100%.

Measures of error and accuracy

Two types of error can occur in surveys: sampling error and non-sampling error. As this was a census survey (i.e. every member of the target population was included), the results are not affected by sampling error. However, census surveys can still be affected by non-sampling errors, which cause bias in statistical results and can occur at any stage of a survey. Estimating non-sampling error can be difficult, so it is important to be aware of this type of error and to either minimise or eliminate it from the survey.

Every effort was made to keep the non-sampling errors in the census survey to a minimum by careful survey design and efficient operating procedures. The following section provides a brief discussion of the main types of non-sampling error that could have affected the census survey and should be considered in making infer-ences to all Aboriginal and Torres Strait Islander APS employees.

Non-sampling error

Coverage error—this error occurs when all relevant population units are not included in the population survey frame. This error applies to the census survey as APSED does not have a complete listing of all Aboriginal and Torres Strait Islander APS employees.⁵⁹ The survey was extensively promoted to encourage Aboriginal and Torres Strait Islander APS employees who were not identified on APSED to register with the Commission to participate in the survey so as to minimise this error.

Non-response error—this error potentially applies to any survey with a less than 100% response rate. How-ever, the high response rate achieved (59%) and the relatively similar response rates across strata indicate that this error was minimised.

Respondent error—this error occurs when a respondent does not correctly answer a question and can apply in any survey. The pilot testing process and the online, hardcopy and telephone support options would have helped to minimise this error.

Coding and processing errors—these errors occur when errors are made in the recording and coding of re-sponses and in data processing. The online survey delivery option, where the respondents themselves entered the data when responding to the survey, would have minimised errors in recording and coding of responses. In addition, identifiable errors made by respondents while completing the survey were removed from the results database; for example, blank responses were generally coded to non-response categories.

Interpretation of scales

Scales were included in any question that required a respondent to measure the strength or level of a theoreti-cal construct. In its simplest form in the survey, a scale asked a respondent to rate the level of importance, satisfaction or effectiveness of various workplace variables on a five-point scale.

The scales used in the survey were generally balanced—that is, they allowed the respondent to express one of the two extremes of view (e.g. satisfaction and dissatisfaction). These scales were also designed with a midpoint that allowed respondents to enter a ‘neutral’ response.

When interpreting scales it is important to realise that there is not an ordinal relationship between points in a scale. That is, the strength of opinion to shift a respondent from ‘neutral’ to ‘satisfied’ may be much smaller than the strength required to shift a respondent from ‘satisfied’ to ‘very satisfied’.

Summary indexes

Summary indexes have been used to assist analysis of results of a number of survey questions comprising several parts. The indexes operate to condense a multiple response question into a single index for compara-tive purposes; for example, in exploring respondents’ overall level of job satisfaction, a question comprising 16 factors was summarised into a single index using a point scoring system. In this way, analysis of the 16 job satisfaction factors can be supplemented by analysis at the summary level.

Coding of open-ended responses

The survey questionnaire provided specified response options for each question. It also included open-ended response options for some questions, which enabled respondents to provide a text response to a question. Open-ended options were commonly provided, for example, as part of a specified response question in the form of ‘other (please specify)’.

Some open-ended responses have been coded to assist analysis. Coding involved, for example, removing irrelevant and incidental comments from statistical outputs as well as counting relevant comments against the appropriate response option.

Size of agency and locality

In analysing survey results, comparisons have been made on the basis of size of agency and locality.

For the purposes of the survey, agencies were divided into three groups: small, medium and large. Small agencies were defined as having 250 or less employees, medium agencies were defined as having between 251 and 1000 employees, and large agencies were defined as having more than 1000 employees.

In analysing locality, employees were divided into four categories on the basis of their workplace postcodes: capital city, metropolitan, rural and remote areas.

These categories were derived from the Rural, Remote and Metropolitan Area (RRMA) classification system.⁶⁰ The RRMA is based on the ABS Statistical Local Areas (SLAs) and combines a distance measure (the midpoint of the SLA to the service centre) and a population measure. It also draws on the Accessibility/ Remoteness Index of Australia (ARIA) in identifying the remote category.

The RRMA approach used to generate the categories is as follows:

Capital city: SLAs within the Statistical Divisions of Sydney, Melbourne, Adelaide, Perth, Hobart, Darwin and Canberra.
Metropolitan: SLAs within the major population regions of size 100,000 and over, outside capital city. This includes Cairns, Geelong, Gold Coast-Tweed, Newcastle, Queanbeyan (part of the Canberra-Queanbeyan urban centre), Sunshine Coast, Townsville and Wollongong.
Rural: The RRMA outlines three categories of rural:
- Large rural—SLAs containing urban centres with a population size greater than 25,000 and less than 100,000. Some examples are Albury, Ballarat, Dubbo, Kalgoorlie/Boulder, Launceston, Mackay, Mandurah and Warrnambool.
- Small rural—SLAs containing urban centres with a population size of between 10,000 and 25,000. Some examples are Albany, Armidale, Broken Hill, Busselton, Coffs Harbour, Devonport, Echuca, Gawler, Gympie, Wangaratta and Whyalla.
- Other rural—SLAs containing urban centres with a population size of less than 10,000 which do not fall into the remote category as defined by ARIA.
Remote: Based on the ARIA categories of remote and very remote (ARIA score >5.80) defined as very restricted/little accessibility of goods and services and opportunities for social interaction.

⁵⁶ Agencies provide employee data from their Human Resources (HR) systems to the Commission, and while the Commission tries to ensure data integrity, it cannot accept responsibility for inaccuracies in the data supplied.

⁵⁷ The return date was extended for all participants to 9 December 2005.

⁵⁸ The population was drawn in September 2005 and this was based on the most recent data provided by agencies, which was June 2005.

⁵⁹ Either through omission in information provided by agencies, or individuals not being asked to report, or choosing not to report whether they are Aboriginal or Torres Strait Islander.

⁶⁰ Department of Health and Aged Care 2001, Measuring Remoteness: Accessibility/Remoteness Index of Australia (ARIA), Revised Edition, Occasional Papers: New Series, Number 14, Appendix B: RRMA to ARIA Concordance, Canberra.