Organisational renewal Home page
> Management Advisory Committee > Organisational renewal > Research methodology > Next: Superannuation
‹ Previous page
Last updated: : 20 March 2003
Organisational renewal
Appendix B: Research methodology
Objectives
The project objective can be found earlier in this report. In broad terms this project was designed to assess staff (at all levels and locations) and agency perceptions about the APS as a workforce.
Scope and coverage
The scope of the surveys was the APS as a whole, both as individual employees (graduates and mature-aged workers) and as agencies.
The target populations varied survey by survey. However, all surveys involved the following agencies:
- APS Commission
- Attorney-General's Department
- Australian Bureau of Statistics
- Australian Tax Office
- Department of Defence
- Department of Education, Science and Training
- Department of Environment and Heritage
- Department of Family and Community Services
- Department of Finance and Administration
- Department of Industry, Tourism and Resources
- Department of Veterans' Affairs.
For the agency level surveys Centrelink was also included.
It is important to note that the selection of these agencies was not random. For this reason the results of the surveys are designed to represent the agencies selected for the survey and generalisations to the APS as a whole may not be appropriate. Ten of the agencies were selected as the relevant agency heads compose the membership of the MAC Organisational Renewal Sub-Committee. Defence and Centrelink were both included where possible to cover more APS employees and represent the issues associated with larger APS agencies.
For the current mature-aged employees survey all current on going employees aged 50 or more years in participating agencies were identified using the APS Employment Database and a scientifically based sample of these employees was drawn.
For the former mature-aged employees survey all ongoing Executive Level and SES level employees who separated from the APS (and the participating agencies) in 2000 to 2001 and who were aged 50 or more years when they left were identified using the APS Employee Database. All of the employees identified were selected to participate.
For the Graduate Employees survey all ongoing employees engaged in a graduate program in 2000 or 2001 and located in participating agencies were identified using the APS Employee Database. All of the employees identified were selected to participate.
Sampling frame
The APS Employment Database (APSED) maintained by the Australian Public Service Commission was used to identify possible respondents to the employee surveys (all the non-agency-based surveys).
During the processing of responses to the project employee surveys it became apparent that in some cases there were discrepancies between the responses furnished by survey participants and the APSED data. Where possible these discrepancies were accounted for during weighting and in the generation of the measurements of sampling error discussed later.
Stratification
The employee surveys went to all the individuals identified using the frame (APSED), except for the current mature-aged employees surveys where a sample of individuals was drawn from the frame.
For the current mature-aged employees surveys the project team was primarily interested in ensuring that accurate statements regarding the overall population of the participating agencies could be made. Furthermore, there were a number of variables for which accurate survey estimates were required. Specifically:
- age (50 to 54 years and 55 or more years)
- location (ACT and non-ACT)
- level (APS Level, Executive Level and SES)
- length of Service (less than 20 years and 20 years or more).
The accuracy requirements vary between the demographic variables listed above and this leads to differing sampling rates of these demographic variables. For example, to gain the same accuracy for estimates for a small population (such as the SES) a much higher sampling rate would be required than for a larger population (such as APS Level staff). In other words, a higher proportion of SES officers were sampled than APS Level employees. This does not lead to a bias in the population estimates because the responses are appropriately weighted to take these differing sample rates into account. See the section 'Weighting and Estimation' for further details.
Stratification is dividing the population of interest on the basis of auxiliary information. The advantages of stratification are:
- The representation of different groups within the sample can reflect the proportions that occur in the population (e.g. 60% male and 40% female).
- Minority groups can be 'oversampled'. This is useful if estimates are required for minority groups.
- The results are more accurate. Sampling error is reduced if similar, in terms of the characteristics being measured, units are grouped together.
- Different selection or interviewing procedures can be applied to the various strata.
- Separate information can be obtained about the various strata.
To simultaneously satisfy the requirements for a number of demographics we created the strata by cross-classifying the demographic variables (for example age by location by length of service by level leading, in the case of the current mature-aged workers surveys, to 24 strata).
Sample selection
To enable us to make sound statistical inferences about all mature-aged workers the individuals were randomly selected from each of the strata. Each individual in the strata had an equal chance of selection.
Questionnaire design
The MAC Sub-Committee for Organisational Renewal provided broad topics that they wished to include in the project in mid-April 2002.
As a substantial number of questions were to be asked only of some officers it was decided to use seven questionnaires for the study. Related questions were grouped to make the project questionnaires easier to understand and complete by respondents.
Two questionnaires were designed for the former mature-aged employees survey of Executive Level and SES level employees that had recently left the APS aged 50 or more years. One questionnaire was tailored to those aged 50 to 54 years and the other to those aged 55 or more years.
Two questionnaires were designed for the current mature-aged employees survey all APS officers that are currently in the APS aged 50 or more years. One questionnaire was tailored to those aged 50 to 54 years and the other to those aged 55 or more years.
Two questionnaires were designed for the agency survey of participating APS agencies. One questionnaire was designed for administration to selected agency heads and the other questionnaire was for the human resource areas of selected agencies.
One questionnaire was designed for the graduate employees survey of recent graduate program participants that were still in the APS.
Though all the questionnaires focused on career intentions of APS employees, their views on the workplace and agency strategies to manage their workforce, the diverse nature of the populations of interest and the differing options available to them required the development of these seven questionnaires. The broad topics covered by the questionnaires are outlined in the table below.
| Topic | Former mature-aged employees survey | Current mature-aged employees surveys | Graduate employees surveys | Agency surveys |
|---|---|---|---|---|
| Demographics | x | x | x | |
| Importance of workplace factors | x | x | x | |
| Satisfaction with workplace factors | x | x | x | |
| Recruitment | x | x | x | x |
| Retention | x | x | x | x |
| Re-employment | x | x | x | x |
| Corporate knowledge management | x | x | x | x |
| Impacts of superannuation | x | x | x | |
| Strategies for managing workforce | x |
A high response rate is crucial to the credibility of any survey. Response rates are affected by a variety of factors, one of these factors being how onerous and time consuming the selected respondent perceives the questionnaire to be on initial inspection. There is often a conflict between the desire to collect as much information as possible from respondents and the aim of maximising the response rate. With this in mind, the questions included in the survey were focused on the main project goals in order to reduce the number of questions, and improve the response rate and thereby enhance the credibility of the survey results.
Another significant factor in determining response rates is the presentation of the questionnaire. The shortened list of questions was presented in a format that would substantially enhance its appearance and minimise the probability of respondents missing relevant questions. Considerable effort was made to ensure that both the wording of the questions and the visual presentation of the questionnaire were of a high standard.
The five questionnaires for current APS employees and former APS employees were tested to ensure the information collected would be valid and useful. Focus groups were carried out in Canberra in mid-May 2002 involving officers from the Australian Tax Office, the Australian Bureau of Statistics and Department of Veterans' Affairs.
The feedback from the focus groups was incorporated into the questionnaires. The five subsequent eight-page questionnaires were approved by the MAC Organisational Renewal Steering Committee and printed around the end of May 2002. The two questionnaires for agencies were approved by the MAC Organisational Renewal Steering Committee in June 2002.
Data collection
A questionnaire and a return postage paid envelope were mailed to every selected current APS employee by the APS Commission. However, as the APS employee Database does not contain the addresses of former APS employees, ComSuper was engaged to provide address details for former APS employees. A questionnaire and a return postage paid envelope were mailed to every selected former APS employee by ComSuper. This ensured address details remained confidential to ComSuper.
All the employee surveys were returned to the APS Commission via the normal mail system via the reply paid envelopes provided.
For the agency surveys all participating agencies were sent a questionnaire for completion by the agency human resource area. The agency head was also interviewed by a member of the MAC Organisational Renewal Steering Committee using an interview pro forma.
To enhance response rates, agency heads were requested to provide a high profile to the survey before the questionnaires were distributed. This was successful with many agencies providing valuable publicity (e.g. notices on intranets, etc.). To further enhance response rates, an introductory letter by the Public Service Commissioner was included on the front of the survey questionnaires. General reminders were sent to all selected individuals prior to the intended due date for survey responses.
The questionnaires for the current and former mature-aged employees surveys and the graduate employees survey were dispatched in excess of two weeks prior to the return date. For various reasons associated with mail processes outside the APS Commission it appears many questionnaires arrived late, or not at all. In order to allow for the late arrival of questionnaires, and the dispatch of replacement questionnaires where notified, the due date for responses to these surveys was extended by two weeks.
Data processing
Data entry was performed by the Australian Bureau of Statistics. The data entry system provided checks to ensure only questions required to be answered by respondents were answered and that some simple logical relationships were followed (e.g. APS employees had to be aged 15 or more years old).
Comments written on the questionnaires by respondents were reviewed by the questionnaire development team and remedial action taken where appropriate. For instance, new categories of response were added to some questions to show small but noteworthy populations that expressed new reasons for leaving or staying in the APS.
Blank responses were generally coded to non-response categories. The exception to this practice was for where responses were needed for demographic items for weighting purposes. Wherever possible, values for missing demographic answers were imputed using the other available demographic information provided in the questionnaire.
Weighting and estimation
It is important to note that the primary purpose of a sample survey (such as the current mature-aged employees survey) is not to gain information about individual respondents. Rather, it is to use the information provided by the respondents to estimate or infer a result for the total population. To accomplish this, weights are assigned to each respondent and applied to each of the respondents' replies. The weight assigned to a respondent can be thought of as the number of units in the population that that respondent represents.
For these surveys the weights were calculated by dividing the population of each strata by the number of respondents to the survey in each strata. For example, if there are 4000 females in the ACT and 200 respond, the weight assigned to each ACT female is 20. This assumes that respondents respond the same as non-respondents for the characteristics of interest.
The detrimental effects of non-response are difficult to quantify. Applying the weighting method described above, the responding persons are used to represent the nonresponding persons. How accurately responding persons represent non-responding persons is very difficult to measure. The published tables are calculated under the assumption that that responding persons answer the same as non-respondents. This should be considered when using the data to make inferences about the population.
Measures of error and accuracy
Total survey errors occur as a result of sampling errors and non-sampling errors. In this suite of surveys however only the mature-aged employees surveys were sample surveys. The remaining surveys went to all the identified respondents.
The following response rates were attained. These response rates exclude responses that were received but were insufficiently complete to provide input into the data generated.
| Survey | Response rate (%) |
|---|---|
| Graduate employees | 62.5 |
| Former mature-aged employees | 56.9 |
| Mature-aged employees | 59.1 |
This level of response is very creditable for a voluntary, mail-based survey.
All agencies participating in the agency survey completed the human resource questionnaire and the agency head interview.
The difficulty with response/non-response rates is that there is no precise percentage which can be used to judge the credibility of a survey. Rather, it is a question of how closely those who did respond to the survey reflect the views and perceptions of those who did not respond. It is not possible to explicitly measure such differences without exhaustive and expensive follow-up action. In this survey, with anonymous responses, it was not possible to identify the non-respondents. However, there would need to be a marked difference in the views of non-respondents to those of the respondents to alter the overall results to any significant extent.
A low response rate can result in bias affecting the results. Accordingly the estimates for those strata with lower response rates can generally be regarded as being less reliable. No information was available to allow estimates of the extent of this potential bias to be calculated.
For the analysis presented in this publication it was assumed that there was no significant bias between those that responded to the surveys and those that did not respond.
As mentioned above, the non-response (approximately 40% for the employee surveys) can lead to unquantifiable biases. Other non-sampling errors can result from imperfections in reporting by respondents, errors made in the recording and coding of responses and errors made in processing the data. No quantifiable estimates are available on the effect of non-sampling errors, including the effects of non-response. However, every effort was made to reduce the non-sampling errors to a minimum by careful questionnaire design and efficient operating procedures.
For the current mature-aged employees surveys the estimates have been derived from a sample rather than a complete count of the population. Therefore, there will be some variability between the survey results and the results that would have been obtained if the entire population had been approached.
One measure of the variability of estimates which occurs as a result of surveying only a sample of the population is the standard error. There are about 19 chances in 20 that a sample estimate will be within two standard errors of the true population value. This is known as the 95% Confidence Interval.
For instance, we are 95% confident that the estimate of the population that would consider rejoining the APS (once they left) is between 49.1% and 55.5% (an estimate of 52.3% and a standard error of 1.6%).
The following table illustrates the worst possible standard errors from the sample design associated with estimates from five key questions in the mature-aged employees survey using the entire mature-aged worker population.
| Question | 95% Confidence interval |
|---|---|
| Main reason for leaving at intended retirement age Estimate | ± 2.4% |
| Main reason encouraging to leave before intended retirement age Estimate | ± 3.2% |
| Main reason encouraging to stay in the APS Estimate | ± 2.6% |
| Main reason that would encourage staying beyond intended retirement age Estimate | ± 3.0% |
| Whether would rejoin the APS after leaving as intended Estimate | ± 3.2% |
Interpretation of scales
Scales are any question that requires a respondent to measure the strength or level of a theoretical construct. In its simplest form in these surveys a scale asks a respondent to rate the level of importance, satisfaction or effectiveness of various workplace properties on a five-point scale.
The scales used in these surveys are generally balanced, that is they allow respondents to express the two extremes of view (e.g. satisfaction and dissatisfaction). These scales are also designed with a midpoint where respondents with a view that is not significant in either direction can enter a 'neutral' style of response.
When interpreting scales it is often important to realise that there is not an ordinal relationship between points in a scale. That is, the strength of opinion to shift a respondent from 'neutral' to 'satisfied' may be much smaller than the strength required to shift a respondent to respond as 'very satisfied' rather than 'satisfied'. Thus the numbering of points on the scale from one to five can be misleading. For this reason it is often preferable to analyse the distribution of responses across each of the categories in order to understand the response pattern.
However, when a summary measure of the distribution of responses is required, such as in this report, the mean of the responses can be used as a proxy. The reason means are not preferable is that this technique generally relies on each response category being assumed to be evenly spaced (ordinal).
When using a balanced five-point scale numbered one to five (as in this report) a mean of 3 generally represents the 'neutral' response. It is important to note that a mean of 3 does not mean the population is not disposed towards the issue one way or the other. Most likely some of the population reported values that were not neutral, but that the number and strength of the population reporting on one side of the 'neutral' point are balanced by the number and strength of the population on the other side of the scale.
In this report a measure of less than or more than 3 indicates the response distribution is biased towards one of the response extremes. The greater the difference between the mean score and the neutral response (3), the greater the bias toward one of the extreme responses (e.g. satisfaction or dissatisfaction).
