F: Embracing adaptive government
Last updated: 05 Feb 2016
This page is: current
The Royal Commission into the HIP found that "the tension between the stimulus objective of the policy, with its concomitant need for expedition, and the energy efficiency objectives of the policy … caused a number of decisions to be made … which unnecessarily exposed workers, particularly inexperienced ones, to an unacceptably high risk of injury or death".
Given the political pressure for speed, which curtailed the time available for program design, the APS should have been aware that it would have to learn lessons along the way. Hanger drew attention to the fact that information from the roll-out of the HIP was not used to inform ongoing management of emerging risks nor to act on them as they manifested. There was no testing of whether the right people and skills were available to deliver the project. No attempt was made to recruit experienced individuals to address the deficiencies.
With respect to the NBN, Scales reflected that the Government leapt to creating a new 'start-up' Government Business Enterprise "that was completely untested and ill-prepared to deliver one of the largest, most complex infrastructure projects in Australian history within a very tight timeframe". Government did not think that it might need to adapt as it moved into a new and untested area. There was too little willingness to try things out at a small scale, to experiment and evaluate and to respond flexibly and expeditiously to emerging problems.
In the HIP public servants did not act in a facilitative manner. They failed to harness outside experience. Indeed, Hanger noted a tendency for public servants to isolate themselves from external sources of advice. Express instructions were given not to consult industry. There was also an unwillingness to engage with counterparts in other jurisdictions: "Curiously, and inexplicably, none of the … officers working on the HIP appear to have liaised with their New Zealand counterparts about their respective energy efficiency programs involving home insulation to share information, experience or alert each other to potential problems". This closed approach also extended to the fundamental issue of properly investigating how States and Territories understood their work health and safety responsibilities.
In a similar vein, the review of the BER found that when school stakeholders were not consulted or authorised to make decisions, schools were placed at greater risk of being left with more expensive buildings that were not fit for purpose. The stark contrasts between the results achieved in different school systems and states revealed how old-fashioned command-and-control public administration (and both Commonwealth and some state governments were guilty of this in their haste to get the program going) can ultimately undermine the delivery of quality outcomes.
The traditional path to developing and delivering government policy is well-trod. It involves developing a Cabinet submission to address a perceived problem, gaining Cabinet approval, having the APS design the program, and using public servants or contractors to roll it out across the nation. There is still a place for such tried and tested methods. They can work effectively for policies that deploy at an industrial scale, involve high numbers of transactions and require high levels of accuracy (for example, collecting taxes or making benefits payments). In other areas of service delivery, a one-size-fits-all approach to delivering a program across the whole of Australia will increase the risks of implementation going wrong. In most instances this will reduce the value-for-money that citizens derive from the public funds expended. Inflexibility reduces choice for program clients and diminishes contestability amongst providers. It is often far better to allow programs to be tailored to the needs of particular individuals or communities.
Public services are becoming increasingly contestable. But while it is now standard practice for a range of providers to be contracted to deliver programs, governments still do not face the discipline of full competition in the marketplace. This makes it critical that they find other ways to learn from the experiences of businesses and community organisations. Too often, governments focus their resources on defending a well-established but outdated approach. Doing things differently is often not considered until things go badly wrong and large sums of money have been wasted.
"You can't do it all at once otherwise it will be a train wreck… We're going to get there by doing small things, deliver them very quickly and then iterating them, changing them, improving them, making sure they actually do meet user needs and that we continue to do so as we move along" (July 2015).
Paul Shetler is the CEO of the Digital Transformation Office
But, a quiet revolution is already underway. Pockets of the APS are experimenting with new ways of delivering government business. A consistent theme in most of these projects is the desire to start small and learn from both success and failure. They involve testing a range of actions, evaluating the results, and then shifting attention and resources towards what works best. Programs move to full scale only when the lessons of demonstration projects have been incorporated. Public servants may deliver the services directly or work with third-party delivery agents, who are provided with greater autonomy in how they pursue agreed performance-based outcomes. Success is achieved through iteration. In short, these areas of the APS are adaptive: they learn and adapt through the process of doing.
Positive findings are emerging from using trial sites to explore different ways of implementing the National Disability Insurance Scheme; from place-based, community-driven initiatives in early childhood; from providing longer-term more flexible contracts to indigenous organisations to deliver services in remote areas; and from promoting consumer-directed services to older Australians who require home care. Lessons also come from some state governments, where this gradual, patient, exploratory approach has reaped benefits in helping families at risk of breakdown. There is a palpable sense of change.
There are four elements to adaptive government: being flexible (paying by outcomes and measuring performance), being experimental (starting early and failing quickly), being facilitative (working with others rather than in isolation), and being agile (learning as you go). Applied concurrently, these elements have the capacity to improve and strengthen the way government designs and delivers its policies, programs and services. I have discussed these in turn below, contrasting traditional practice with adaptive practice, and drawing on examples from Australia and overseas.
|Traditional government||Adaptive government|
The starting point for any new project should be developing a deep understanding of the objectives pursued by government. The opening question should be "what will success look like?". It is necessary to identify the results that are sought, and agree (in collaboration with potential providers) the outcomes against which performance will be measured. The Enhanced Commonwealth Performance Framework, introduced under the PGPA Act from 1 July 2015, emphasises the importance of Commonwealth entities reporting on outcomes and impacts in their corporate plans and annual reports. The new corporate planning requirements encourage agencies to identify what success looks like at the beginning of the annual business cycle, and to explain how they will measure it over the short, medium and long term.
The APS is still a long way from being able to do this consistently. Too often public servants are reduced to monitoring processes, ticking off programs against a series of outputs, acquitting payments or meeting contractual conditions. Meanwhile the ambitious goals of government and interests of the citizen become lost in the mechanics of grant applications, contracts, guidelines and reporting rules. There remains too much focus on process (how many clients have been seen) and outputs (how many services or payments have been delivered) rather than on outcomes (how has welfare dependence been reduced, employment increased or health and well-being improved). Unfortunately, the innovative impulse of delivery partners is too often stymied because APS contract managers restrain unnecessarily the approach they can take. Contracting is transactional, with management focusing its attention on legal and procedural compliance. In contrast, commissioning of service delivery undertaken in a flexible way can be transformational: management can direct its attention to performance.
One way to promote more flexibility in program delivery is to shift to Payments by Results (PbR). It should not be underestimated just how challenging this can be. PbR requires pre-agreed measures to be established. Payments are contingent on the verification of results, some of which may become apparent only over the medium term. Counterfactual approaches are required, allowing estimates to be made of what results would have occurred over time in the absence of an intervention. The better the quality of the performance metrics, the more discretion that can be allowed to front-line public servants or contracted providers in how to deliver them.
Of course, outcomes need to be set to discourage gaming. Experience warns us that simple targets often result in behaviour that weakens the underlying purpose of the program. Incentivised performance measures, in themselves a useful feature, can sometimes distort provider behaviours. Nor am I advocating a laissez-faire approach, allowing contractors to claim that virtuous ends justify dubious means. Nevertheless, the benefits of applying a PbR approach methodically more than outweigh the up-front costs invested in its development. Payments can be made on the basis of the benefits that are being delivered. Through upfront agreement on outcomes, and the removal of restrictive controls, public sector entrepreneurship can be liberated.
The flexibility engendered by PbR approaches can go much further. An example that I have been intimately involved with is the introduction in Australia of Social Benefit Bonds (in the UK, Social Impact Bonds). They represent a financial instrument that pays returns to investors based on achieving agreed public outcomes. This approach has been trialled in NSW to deliver better results for families at risk of breakdown. The programs to reduce levels of out-of-home care have not been designed by public servants but by innovative community organisations (UnitingCare and The Benevolent Society). performance-based outcomes have been negotiated, and funding raised from the private sector. The providers carry the risk but they are given the freedom to deliver the program as they wish. Such public-community-private partnerships (impact investing) can have much wider application. The Joint Standing Committee on Foreign Affairs, Defence and Trade has recently recommended assessment of Development Impact Bonds as a means to improve the effectiveness with which overseas aid is delivered.
|Traditional government||Adaptive government|
The biggest impediment to greater use of experiments in the delivery of public services is risk aversion. For this reason the adoption of a positive risk culture is a critical first step on the path to more experimental approaches. A positive risk culture, as has already been discussed, involves public servants accepting responsibility for risk at all levels of the organisation, and allowing risk management and decision-making to be devolved to those who are close to the action.
It is useful on occasion not to 'think big'. For initiatives that are entering relatively new areas of government policy, perhaps we should be willing to countenance giving public sector managers the licence to 'start fast, test first, fail small'. After all, why can't governments on occasion act more like businesses and be able to prototype and trial delivery models and learn from these experiments before proceeding to scale? This is not about wasting public money on administrative fads or bureaucratic fopperies. Rather, it is about testing the most efficient and effective ways of delivering on the ambitions of government. Establishing clear boundaries from the top down as to what risks can be taken, and in what circumstances, will allow public servants to establish the best approach.
Trialling before moving to full scale does not mean delayed implementation. I have no desire to sign up the APS to the 'Slow Movement', nor, conversely, to promulgate haste. The key to good public administration is to do everything at the right speed. It is often faster to deploy a single prototype from which to learn, than to design and deliver a full-scale program that is likely to be beset with problems from day one. Better still, demonstrations allow public servants to compare the efficacies of different approaches. On occasion trials may suggest a need for quite different methods of pursuing government policy, but in most instances they will allow public servants to identify a plethora of incremental improvements that, cumulatively, can drive significant efficiency gains.
Trials should not be limited to new programs. Opportunities are also available to improve large, ongoing programs. For example, the Behavioural Insights Team, formerly based within the UK Cabinet Office and now a company operating at arm's length from government, has carried out extensive randomised 'nudge' trials to test end-users' responses to a range of prompts. This testing was undertaken at low cost and quickly showed which approaches worked best. Subsequently, the UK Government was able to implement successful policies that achieved improved outcomes (such as increasing organ donation by 100,000 people in a year, and lifting tax payment rates by 5 per cent)  A similar White House Social and Behavioral Science Team has been established to help the United States Government identify approaches that harness public behaviours to improve effectiveness.
Understanding the psychology of market interventions is just as important as understanding the mechanics. The most elegant policy solutions will fail if they do not account for how humans behave. As Allan hawke noted in his review of the HIP, the program design provided little incentive for householders to think about the quality or performance of insulation installers. This goes to a more general point: there should be an expectation that public servants, when designing policy and programs, have made themselves aware of what has been done by others, what has worked well, and what has not. Public servants should take the time to 'stop and look around', not just at the operation of markets, but at the behaviour of people.
The good news is that this focus on the social, cognitive and emotional behaviour of individuals and institutions, often explored through testing different approaches, is now being taken up in Australia. For example, a Behavioural Insights Unit has been established in the NSW Department of the Premier and Cabinet. It collects data from the front-line and then undertakes randomised controlled trials, incorporating results into the design of government interventions. A small but growing number of Australian Government agencies is also beginning to apply or trial the use of behavioural insights techniques. Greater investment in these approaches, both inside government and in think-tanks, can only help spur the evolution of richer insights. There would be considerable value in the APS fully engaging with the Behavioural Insights Community of practice that has been established in Australia and, better still, creating a similar group of its own. It would act as a catalyst for experimentation.
The APS can learn much from Australia's large publicly-listed companies about experimentation. By virtue of their size, these companies have many of the characteristics of bureaucracies. They are generally demarcated horizontally by function (into subsidiaries or business units) and vertically by hierarchical reporting lines. Committee structures dominate the processes of decision-making. From the outside—and as I now know, from the inside—they look very similar to large departments of state.
The difference is market competition. Although they enjoy the considerable power of incumbency, each company recognises that its continued success depends on addressing the constant threat of disruptive innovation. Challengers, from large global multinationals to small entrepreneurial start-ups, constantly improve the price, quality or consumer attractiveness of the goods or services offered for sale and the manner in which they are produced and distributed. Organisational sclerosis means certain decline. Sometimes companies die slowly by a thousand competitive cuts; occasionally they collapse precipitously in the face of dramatic shifts in consumer preferences. Loss of market share, declining profit and susceptibility to takeover are ever-present dangers. Many of the great companies of the past, household names for a generation or more, no longer exist.
Each company knows that it must find ways to maintain and grow its shareholder value. Each, led by a CEO and executive management and governed by a chair and board of non-executive directors, will develop its own strategies. There is a discernible pattern of good practice, however, from which public service agencies can learn.
A forward-looking company will seek to do more than react speedily to emerging threats. It will set its sights on identifying untapped opportunities that can provide it with a competitive edge by which to deliver sustainable shareholder return (in contrast, the driving force for a government agency will be to deliver public outcomes better). It will often trial different approaches to demonstrate which is likely to be more effective. Some initiatives will then be deployed at scale, others extended more broadly, and others shut down or sent back to the drawing board for further work. The core systems of the company will be adapted to ensure that they can support the new approaches or products. Often that involves motivating intermediaries that stand between the company and its ultimate customer. Not all initiatives will prove successful: behavioural psychology is complex, and consumers will often respond in unexpectedly negative ways to change. Some interventions, by contrast, quickly deliver tangible improvements. A well-managed company learns from both. It is a form of disciplined entrepreneurship. It is experimentation, driven by purpose. The APS should embrace such approaches.
|Traditional government||Adaptive government|
The APS will continue to sit at the centre of public administration. Its senior leaders have extensive access to ministers' offices. They participate in the confidential meetings that discuss new policies. Relations between the government and most of the lobbyist or advocacy bodies with which it meets, and the multifarious organisations which it regulates or influences, are to a large extent conducted through government agencies. Public servants, directly or by contract, deliver government services to the public. They are the means by which the entitlements and obligations of citizens are communicated.
What needs to change is the ethos that is brought to that situation of positional authority. Public servants cannot seek to be controllers. Indeed, the PGPA Act establishes a duty in law for officials to co-operate with others to achieve common objectives. Adaptive government depends upon them exercising their responsibility on the basis of collaboration and partnership, working cooperatively across sectors to inform and deliver a government's agenda. They need to see themselves as the stewards of democratic processes and good governance. The leadership they provide needs to be facilitative in nature. Their performance should be assessed on their ability to effectively harness ideas and capabilities from across and outside of government, not on their ability to control and orchestrate every minor activity.
Let me provide two instances of changes already underway. Both call for public servants to apply high-order facilitative skills. The first example builds on almost two decades of experience in brokering the delivery of Australian Government labour market programs on a competitive basis to a range of public, private and community providers. As Secretary of the Education Department, Science and Training in the second half of the 1990s, I remember well the introduction of the Job Network (now jobactive) and the exciting prospects that it held. I envisaged a public economy in which job seekers would be able to select the organisation that they wished to deliver the training or labour market support that government funded.
That ambition has been only half-fulfilled. The goal seemed so straightforward—setting an outcome (how many jobseekers were placed into work for 13 or 26 weeks) and allowing contracted providers to decide on their own approach to achieving it. Unfortunately, the process has become burdened by tomes of prescriptive guidelines. The organisations have been treated as if they were mini-government agencies and expected to do things in very similar ways.
It does not have to be like this. The Department of Employment is seeking to modify its approach. By collaborating with potential providers on policy design and execution, and paying on the basis of performance-based results, a greater spirit of partnership can be created. providers can be given more flexibility. Of course, it will still be important for public servants to ensure that providers behave honestly and ethically. Between the co-production of policy design and careful monitoring of outcomes, however, perhaps the best role of the APS is to stand aside and let the organisations commissioned to deliver the services get on with the job.
Martin Parkinson PSM
"There are expectations on us to engage differently with business and the broader community—to better understand and incorporate their perspectives into our policy analysis and development." (March 2014) 
Martin Parkinson was the Secretary of the Treasury from 2011 to 2014
The second example of working with others in an adaptive way is to go beyond 'customer service' and allow those who receive services to wield more control. The emerging commitment to consumer-directed care allows those entitled to government services to manage a care budget and make their own decisions. From July 2015 this year, for example, Home Care Packages for the elderly provide individuals with the option of tailoring care to their particular needs, with the assistance of service providers. For public servants this will involve a much greater need to work not only with a range of contracted providers but with the individual citizens who will access their services.
Such initiatives offer great opportunity for government to actively encourage consumer or community choice and then to learn from the preferences that are revealed. This information can improve the design of public services—just as any company in a new market carefully monitors and reacts to how its customers respond to its products. Actively supporting community organisations or individuals to be engaged in service delivery also helps build their skills, resources and social capital, allowing them to independently generate beneficial public outcomes.
Business has been quick to harness the power of digital platforms to use their customers' views to shape their product offerings. Governments in Australia still lag behind. Done well, the techniques of digital democracy can help to drive citizen engagement in ways that can help shape government policies. Of course, the technology is just a tool. Real power lies in the willingness of public servants to employ a range of facilitated deliberative processes to involve citizens in reframing questions and suggesting answers. Simply signing up to social media will not change things unless the APS is ready to take on lessons from outside the public sector.
|Traditional government||Adaptive government|
Three things can stymie effective sharing of lessons across government: the functional demarcations of bureaucratic structure; an unwillingness or inability to discuss failure; and the over-use of confidentiality and security provisions as an excuse not to consult broadly. The effect on APS capability is clear. Less obvious is the manner in which discussions with business and community organisations and the wider public are impoverished. It lessens the opportunity to hear from those impacted by government programs, and reduces the ability to evaluate the impact of government investment. This is not just an Australian Government problem. In spite of the occasional nod to the potential benefits of co-operative federalism, Australian jurisdictions have historically been poor at sharing information about different policy approaches between levels of government.
For agencies to become learning organisations they must transition from a 'need-to-know' approach to information to a 'need-to-share' philosophy. Of course, standards around security, confidentiality and privacy cannot be compromised. But the APS needs to find ways to improve how it shares data and experience from early lessons, rather than relying on post-implementation audits and risking large-scale failures. In 2008 the Venturous Australia review, conducted by Terry Cutler, recommended that "to the maximum extent practicable, information, research and content funded by Australian governments… should be made freely available over the internet as part of the global public commons". Since 2013, the Australian Government has required the results of research funded through the Australian Research Council to be openly available to the broadest possible audience. Perhaps this requirement could be extended to all government funding, similar to many areas of public funding in the US. Perhaps too, we might learn from the UK Government, which regularly creates opportunities to assess and publish the results of different approaches to government delivery.  Staying agile, continuously learning and adapting requires a change of mindset.
"You can say government needs to be more cautious and that's true, but it's not an excuse for failing to be more agile and operate in a different way… You have to ask 'what would government services look like if Apple or Google delivered them?' and you get a different answer to what we see in the public service." (2014) 
Marie Johnson was the Chief Technology Architect for the Department of Human Services from 2011 to 2014
Contracting out service delivery has not delivered on the promise of flexibility and choice. The failure speaks to a larger issue. The contestability agenda is too often conceived narrowly as a drive to outsource service delivery to the private sector or communities at the lowest cost (which is how value-for-money is all too often perceived). This misses the larger purpose of identifying the most effective and efficient ways to achieve a government's desired outcomes. Contestability should introduce credible competition between implementation options based on benchmarking and market testing. A diversity of approaches should be actively encouraged so that delivery is undertaken by a variety of providers in different ways. Improvement can be informed by monitoring the experiences and evaluating the outcomes of those 'doing the doing'. Just as businesses learn by scrutinising what works for their competitors, so too can the APS learn from studying the providers of public services. It can acquire knowledge of good practice and innovative approaches by assessing their performance. The key is to focus on results, learn from experience and to stay agile in the search for innovation.
The best way to discover the value of adaptive government is to do it. Pay on outcomes, embrace experimentation, start early and (if things go wrong) fail quickly, collaborate widely, and learn as you go (including from the experience of others). Adaptive government can take many shapes. There are innumerable ways in which the public service can become more agile. As with any new approach, changes will be needed to turn the adaptive ethos into action. Some will require a shift in entrenched practices, attitudes or behaviours. Administrative structures and workplace systems often hinder the capability of APS agencies but it is cultural inertia that acts as a barrier to creativity. Induction programs generally focus on explaining to newcomers that "this is how we do things here" rather than "this is how we are looking to improve things here".
There are disincentives for ministers to propose adaptive approaches, even if they would like to do so. The large number of competing priorities at Budget time creates an incentive for ministers and their departments to downplay the costs and risks of new initiatives and talk up the benefits in order to secure larger expenditure commitments to their portfolio. This in turn creates an incentive to adopt and promote expansion of a well-tried implementation approach, rather than admit the wisdom of trialling and demonstrating a range of approaches (some old, some new) to achieve the outcomes sought. Would it not be advantageous if, when bringing forward a new policy proposal, ministers should have to justify why they should not have to start by piloting their idea on a small scale and then, on the basis of trial and error, prove their implementation strategies on the ground before it is decided to spend large sums of money? For adaptive government, at least initially, small is beautiful—it is a clever way to do big things better in the future.
"Governments often do slowly what should be done fast, and fast what should be done slowly. Ill-thought out reforms are rushed into implementation at great cost. The experimental method offers a reasonable compromise—fast action, but on a small scale, leading to phased adoption at a larger scale. That gives politicians plenty of examples to point at, but at less risk." (2015)
Geoff Mulgan was Director of the Prime Minister's Strategy Unit and Head of Policy in the Prime Minister's Office, United Kingdom, from 1997 to 2004
Adaptive government necessitates other changes. The assurance reviews framework administered by the Department of Finance requires proposals that are high in risk, priority, value and complexity to be assessed by independent experts. Whilst the framework has raised delivery performance, there is potential to make far better use of its Gateway Reviews and Implementation Readiness Assessments (IRAs). Feedback from those involved in the reviews indicates that they are helpful in improving the management of a large program or project. Experienced practitioners comment that it is usually obvious very early whether the initiative is likely to succeed or fail.
There may be scope to put a broader range of proposals through the initial stages of the Gateway process, while allowing initiatives that perform well in the early review stages to opt out of the later ones. Similarly, there is scope to be bolder with assurance reviews, using them to assess whether to stop doing something that is not going well, rather than just offering suggestions for improvement. Those involved in reviews are sometimes frustrated that the Gateway process misses the mark on outcomes. Too often the process is geared around checking that the proponents "are doing things right" rather than that they "are doing the right thing". There is little value in agencies demonstrating best practice process when there exists a fundamental design flaw that means that implementation can never properly meet the outcomes sought by government. Building greater flexibility and adaptability into the assurance reviews process would remove unnecessary red tape while making scrutiny more meaningful.
In some cases, staged financing creates perverse incentives to move too quickly to scale. The typical approach is to set aside the full project budget in the contingency reserve, and to release it once the business case has been established. This creates an incentive to talk up the business case in order to secure release of the money. A preferred alternative might be to provide sufficient 'seed capital' to allow projects to complete not only their business need and business case stages but also to develop a proof-of-concept. This evidence could then be submitted to make a case for funding a full-scale project, although it might sometimes reveal that the project is not likely to work as conceived and that further funding should not be provided, at least until other approaches have been considered. Linking this 'gated' funding process more strongly to the Gateway Review process would allow funding to be withheld until independent assurance is received that the project is going well. It might reduce the likelihood of throwing more money in the wrong direction.
The manner in which governments select modes of delivery also needs to change. New Policy Proposals (NPPs) have a laudable focus on providing evidence that supports policy intervention, but traditionally this has included little evidence about the mode of delivery. The implementation of major new projects and programs should form an equally important part of the evidence base for decisions by Cabinet or recommendations of the Expenditure Review Committee. Assessment of whether the proposed delivery methodologies have been applied or trialled successfully in the past will assist ministers to consider execution risk. Too often there is no effective pathway to gather evidence from stakeholders—the organisations, communities and individuals who can contribute practical suggestions on how to deliver new proposals most effectively. They may know much better than APS senior management what will work on the ground.
Government, through skilled public service facilitators, should actively solicit concrete proposals from the private, community and academic sectors on how their programs could be delivered better. Perhaps outcomes can be better defined; or cost savings redirected to improve services; or the burden of public service regulation lessened; or citizens given more opportunity to make choices on their own behalf. The possibilities are limitless. Opportunities should be introduced that allow government to proclaim its adaptability.
An annual, well-publicised competition might be held to gather good ideas from business and the community on how to improve the delivery of the government's major programs. For example, a Chamber of Commerce might design a scheme to encourage businesses to hire older workers and organise supportive companies to test different approaches. A group of doctors might undertake a small trial of different ways to deliver preventative health education and share the results. An emerging tech company might submit a 'Trip Advisor' type application to help people navigate the range of disability services available or to select their own aged care provider. Here is an opportunity, on a grander scale, to build on the successes of GovHack. A small amount of money could be awarded to set the winners on their way, and public service mentors selected to assist them negotiate the labyrinthine systems and processes of government. Where the proponents can test the concept themselves, lend support. Where the idea would be best tested by the APS, the proponent might receive a cash prize, and (probably more important) a commitment from government to involve them fully as the test proceeds. For all ideas, the results should be published. Additional funding should be considered as part of the Budget, if the results of the test turns out to be encouraging and useful. The winners would gain not just a prize but public recognition. Equally important, the government would show itself to be actively encouraging and welcoming innovation.
Adaptive government can go further. The full benefits of contestability will not be realised if contracting continues to be designed to encourage all providers do their business in the same way. At present, the APS tends to default to process-driven, output-based and highly prescriptive contracts, elaborated in voluminous administrative requirements. This does not allow sufficient latitude for third-party providers to explore and implement better approaches. Opportunities exist for APS leadership to endorse a more fulsome exploration of the different funding models available by which to contract performance-based outcomes.
There are already established models available, such as alliance contracting, performance contracting and using schedules of rates and cost-plus models rather than lump sums. These models balance the probity and accountability that citizens expect to be applied to public money, with harnessing external expertise in the design and implementation of policy. As well, they create more opportunities for ideas on program design to come from outside the APS, and allow risks and benefits to flow freely between delivery partners. In essence, this is the approach which underpins the introduction of Social Benefit Bonds in NSW. What matters is that contracts provide both the flexibility and the incentive to develop and trial ideas. Innovation can improve results. These different forms of contracting need not be more risky. Rather, risks can be negotiated and managed by the party best placed to do so, with provision made to share both the upsides and downsides of experimentation. Commonwealth contracts can be written and managed to encourage public entrepreneurship.
The Government's strong commitment to reducing red tape and making it easier for the public to deal with government must continue. This requires more than removing or simplifying regulation through the legislative process. It requires positive engagement with business, the community and citizens to better understand how their lives can be made easier when working with government. This will not occur, for example, if contracts continue to include onerous reporting conditions or grants programs have overly prescriptive guidelines. The APS needs to value the time of its delivery partners. The Government will benefit from lower prices as partners reduce the premium they build into Australian Government contracts in order to deal with the cost incurred by poor administration. During consultations for this review, it was suggested that some outsourced providers may add up to 25 per cent to prices to cover the cost of the transaction. Here, often hidden from view, is the true burden of red tape.
Reducing the administrative and reporting guidelines imposed on third-part agents does not mean licencing delivery partners to do 'whatever it takes', or turning a blind eye to risky or inappropriate business practices. Rather, working in partnership to simplify dealing with government should lead to better risk management, through better understanding of roles and responsibilities, greater trust between partners, and greater accountability.
But wait, there's more. Many of the excellent ideas of the Government 2.0 Taskforce from 2009 have largely fallen by the wayside or have emerged as reporting obligations rather than opportunities for change. Recent surveys indicate that Australia is yet to fully translate its high-quality digital infrastructure and human capital into high-quality online service delivery. The 2014 United Nations E-Government Survey indicates that while Australia rates highly for service delivery (8th internationally), in the provision of connected services (that is, citizen-centric online solutions that cut across departments to allow data transfer), Australia rates 65 per cent where leading countries are closer to 100 per cent. Compared to the UK and United States, Australia's progress on open data policy implementation has been fragmented and lacks sustained conviction. As a consequence, government is less adaptive.
As part of this process, the APS should open up to new forms of citizen engagement both through increased sharing of government data and information and by providing more online opportunities to participate. Technology solutions, including digital democracy, can empower citizens to exercise the greater discretion they are being given. There is an increasing interest in many Westminster countries in 'citizen-centred governance'. It involves finding ways to devolve power and influence to citizens, communities and service-users. In the UK the approach has become a key component of government policies to tackle social exclusion and welfare dependency. Implemented well, citizen engagement can improve the design and responsiveness of services, build social capital, encourage civic participation, and build greater trust in democratic institutions. At the local level, where there is opportunity for place-based solutions, citizens can attend in person with discussions facilitated by public servants. At the regional or national level, in most instances, engagement will have to be elicited online through webinars, chat rooms, deliberative polls, or structured (but open-ended) questionnaires. Think of it as digital democracy with a purpose. As was recognised in Western Australia, 'e-engagement' allows individuals to participate who would usually not be interested in traditional methods of consultation.
"I obviously realised very quickly if I come in as the subject matter expert telling career and life-long bureaucrats about how to do their job, I was probably going to have a short lifespan. What I did have… was a set of skills that they really needed… [a] modern approach to customer delivery… a lot of technology background… a real commercial approach… I then used these skills in a way that was clear to the Directors-General and their teams that I was there to help them. The first months for me were basically active listening." (August 2014) 
Michael Pratt is the Services NSW Commissioner
Such notions of active citizenship are not new in Australia. Nor do they have to be online. I have witnessed first-hand a range of projects that have been designed to bring members of the public together to discuss common concerns and to participate in the decision-making that affects them. The citizens learn negotiation skills, not helplessness. The public servants learn to listen. The design of programs may be improved and, in most instances, the manner in which they are delivered is enhanced. Of course there are hurdles, from the identification of stakeholders to agreement on purpose. Experience shows that problems can emerge during the process, ranging from collaboration fatigue to aggressive behaviour.
Yet it is disappointing that such citizen-centred approaches—which would have been so useful to informing home insulation or school hall construction—have so rarely been built into major programs. One key reason is that the expenditure of time and resources required continues to be seen as a cost rather than an investment in improved public benefits. Such perceptions need to be turned on their head. By involving the community early in planning, it is likely that programs can be delivered at lower risk and provide greater value-for-money.
If the APS invests in modernising the online engagement skills of staff at all levels, encouraging them to explore in the virtual world and discover innovative ways to engage with the public, the emerging techniques of digital democracy are more likely to become powerful tools for experimentation. Many options are available. 'Ideas markets' can allow government to be informed by votes from the community. Text mining tools can make assessments of a large volume of online submissions. Open publishing of government data, such as data.gov.au can allow third parties to make new uses of government data sets. Citizen juries or online town hall meetings can be effective tools to help build consensus around difficult choices within constrained budgets. The internet can encourage the public to report fraud, waste or emerging implementation problems early. New citizen spaces, such as, 'We Asked, You Said, We Did' in the UK allows governments to be more sensitive to market responses in the same way that successful companies are.  Such approaches, already trialled by the South Australian and Western Australian public services, should be embraced by the APS. Deliberative technology is becoming even more sophisticated, enabling citizens to contribute to political decision-making in structured ways. The new Digital Transformation Office promises to make digital delivery of government information and services the new default. This is a vital and long overdue step in meeting the expectations of citizens regarding how they engage with government.
The APS needs fully to comprehend the extent to which the public is attracted to the concept of public service. GovHack has just celebrated its fifth year. It has grown from a small mash-up event in 2009 to a huge competition that brings together large numbers of enthusiastic volunteers in 30 locations to innovate, collaborate and apply their creative skills to open government data. Some of this entrepreneurial innovation is applied directly to improving the way in which government services are delivered. Much of the energy goes into developing new apps for the community based on government data. All of this boundless enthusiasm has the potential to contribute to the creation of beneficial public impact. The event exemplifies the ethos of structured experimentation that should imbue a contemporary public service and adaptive delivery. It requires openness to different ways of doing things and active efforts to maintain goodwill with participants, by ensuring that the format remains fit-for-purpose, and is engaging and rewarding for those who take part.
Martin Bowles PSM
"Be stewards of the system you're dealing with; don't try to be the owner." (July 2015) 
Martin Bowles is the Secretary of the Department of Health
Adaptive government must make full use of the potential of digital democracy. It will make it easier, cheaper, less risky and quicker to bring to government the experience and knowledge of businesses, not-for-profit organisations and universities. It will enable tech-savvy citizens to involve themselves in the full gamut of policy development and delivery. Major programs and projects should incorporate transparent and responsive digital engagement with citizens from the earliest design stages through to operation and completion. This should be done not just because it is more democratic, but because engaging widely is often the way to generate more insight, quicker. It can tap into and connect sources of expertise and experience that are distributed across the public.
The term adaptive government may be new. Its underlying principles are not. "The future is already here", said the science fiction writer William Gibson, "it's just not evenly distributed". That insight sums up the state of Australian public administration. To build and sustain innovative approaches the APS must become more open to outside ideas, learn from business, value community experience, sponsor academic research and 'crowdsource' citizen proposals. This commitment to an outward-looking APS needs to be championed at the highest level of government and public administration if it is to provide a strong foundation for tackling future challenges.
The factors contributing to more open and flexible government have been the subject of rigorous discussion for many years. There is evidence that they deliver results. Why, then, do these approaches so often remain at the fringes of public administration? Part of the problem, perhaps, is that those at the centre of government perceive that their position will be undermined should they be open to new voices, to admitting and discussing failure or to explicitly seeking advice. Such fears are misplaced. Nor are they universally held. From what I have seen in the course of this review, many parts of the APS are well-positioned to get on with the business of implementing adaptive approaches to government. They are prepared to innovate, manage the risks, learn from experience and be held accountable for the results. They are waiting for permission to start.
If adaptation and agility are to become widespread practice, the roles of the public servant and the minister will need to change. Instead of controlling the whole process of implementation, they will act as stewards, shepherding the limited resources of government towards a successful result. Humility will be needed to accept that the shape of policy, or at least the way it is implemented, may evolve in unexpected ways in response to evidence and experimentation. This will be a challenging shift for those who feel more comfortable with a command-and-control style of leadership. The public servant of the future will be the facilitator of innovation.
It will not be sufficient for leaders to set managers 'free' to be adaptive. Capacity and capability need to be enhanced. Often governments that have begun experimenting with adaptive government have found their aim of 'transforming markets' has faltered because it was not sufficiently complemented by investment in new human resource skills. Nor is being adaptive just a top-down process. Some of the best ideas to improve delivery are languishing at the front line, lacking a channel to those who have the authority to adopt them. In some areas poor policy design has generated great innovation in delivery as frustrated front-line staff come up with work-arounds and improvised solutions in order to get things done.
Taking the first step requires trust. Departments must trust that their ministers will back them, so that they can learn from mistakes. Ministers must trust that citizens have the common sense to see that it is smarter to 'fail fast, fail small' rather than to pretend that failure is impossible. Citizens must trust that government will learn and improve, and that an unsuccessful trial is not a waste of public resources. Being agile needs to be authorised. Leaders should take heed of the words of the former premier of NSW, the Hon Barry O'Farrell, who extolled the value of public servants thinking differently, even if on occasion failure was the result. As he said in a 2013 address to public servants, "[I]f you are being innovative, and from time to time there are failures, don't expect the Premier …. to give you a hard time … if your goal was the correct goal. Because stuff ups do occur, mistakes do happen—that's why we trial things, that's why we undertake pilots. But unless … we have the courage to innovate, unless we have the courage to think about how to do things differently, we won't deliver the excellence that I'm determined to—through you—across this state."
There is deep cynicism in some quarters around trials and pilots. Some of it is well-founded. It is unfortunate that trials have often been used in the past to fob off interest groups or to avoid committing the level of resources necessary to tackle a hard problem. These are trials employed as cost-saving measures rather than as a demonstration of effectiveness. On the other side, ministers can be too eager to move to full scale at the first sign of success in a trial. It takes time to understand why a trial succeeded in one instance and whether this result can be repeated. Often success is based on particular factors—often, I discovered in Indigenous Affairs, the leadership of a particular individual in the community. Moving too fast inevitably results in disappointment when benefits are not replicated. Once a decision has been made to grow to scale, a measured process is required to get there. Trials can also be seen as an impediment to "getting on with the job". There will be times when government needs to move very fast and won't have time for formal trials. At these times the adaptive approach comes into its own: intensively monitoring, adjusting and evolving policy roll-out reduces the risk of a fast-moving policy coming off the rails. Here is a good opportunity to test and refine the implementation process. Otherwise, as the old saying goes, there will be more haste and less speed.
Government must be genuine, open and honest about its reasons for trialling and piloting programs and the timeframes within which they will take place. Announcing a trial need not communicate a lack of commitment to following through on a policy. On the contrary, it should mean that government is so committed to achieving a successful outcome that it will carefully investigate the best way of doing so. Announcements will be less definitive, more open to possibilities and, at least initially, involve smaller expenditures. There will be more opportunities to talk about progress along the way. Major programs will increasingly begin in minor ways.
Not every area of government operations will benefit from such an adaptive approach. It is not a panacea. Adaptiveness is better suited to areas in which problems are complex, uncertainty prevails, risks of failure are high, and there are a range of possible options for intervention. Governments may find it challenging to invest the 'patient capital' to build new programs that move to scale slowly. Patience, however, can provide its own very substantial rewards.
In a world with 24-hour media cycles driven by 'gotcha' moments and demands for greater accountability, it is difficult for government to admit failure. But to presume that no public servant will ever make a mistake is hubris. I made many and I've reflected publicly on why I did. Accountability means being honest about the limitations of what one knows, and having the courage to admit to a mistake and learn from it. For government to become more adaptive, it will be critical to proclaim this message over and over: that ministers and senior leaders who are truly accountable are those who admit to making mistakes, and can demonstrate that they have learned from the experience.
It is a great shame that public trust in government institutions is not greater. It helps to explain the even more worrying fact that young Australians are losing their faith in the benefits of democracy. Adaptive government can provide a practical mechanism to rebuild this trust, manage risks and demonstrate that government is both learning from the past and responsive to the national challenges of the future—in partnership with citizens. Perhaps the best way for the APS to honour the lessons from the HIP, and the failure of many other major projects over the years, is to demonstrate that change has manifested itself in a new culture of public service. Out of tragedy let there be transformation.
CONCLUSIONS | Embracing Adaptive Government
F.23 The default position that new policies proceed straight to large-scale roll-out should be reversed and instead new policy proposals should include a trial or demonstration stage, allowing new approaches to be developed fast and evaluated early.
F.24 Staged decision-making for large projects should incorporate the allocation of seed funding to agencies to develop a business case and proof-of-concept, which can be tested before the project moves to a further stage.
F.25 The Australian Government should fund an innovation competition to encourage experimental, innovative community and business proposals for improving the delivery of programs and services.
F.26 In order to improve contestability and citizen choice, departments should facilitate the ability of contracted providers to take their own approaches to the delivery of agreed performance-based outcomes.
F.27 As part of continuing effort to reduce red tape, greater efforts need to be made to engage with communities and businesses to understand how contractual conditions and administrative guidelines can be less prescriptive, making it easier to work with government.
F.28 The APS should promote new forms of civil participation, including digital and deliberative democracy techniques, in order to enhance consumer-directed care, improve customer service, encourage greater citizen engagement and inform the public economy.
 Hanger, I 2014, p. 2.
 Hanger, I 2014, pp. 209, 309.
 Hanger, I 2014, p. 300.
 Hanger, I 2014, pp. 299-300.
 Scales, B 2014, p. 80.
 Hanger, I 2014, p. 83.
 Hanger, I 2014, p. 97.
 Hanger, I 2014, p. 234.
 Australian National Audit Office 2010, 'Building the Education Revolution—Primary Schools for the 21st Century', Audit Report No. 33 2009-10: Performance Audit, Canberra ,<http://www.anao.gov.au/uploads/documents/2009-10_audit_report_33.pdf.>
 Pearce, R 2015 'DTO chief: Australia could be 'best in the world' for digital government'Computerworld 31 July; Thompson, P (2015) ''We are failing': Digital Transformation Office CEO Paul Shetler warns public service', Sydney Morning herald, 28 July.
 An independent review of the NDIS has been commissioned by the Department of Social Services, <http://www.ndisevaluation.net.au/>. Information on the National Evaluation of the Communities for Children Initiative is available at Muir, K et al 2010, 'The national evaluation of the Communities for Children initiative', <https://aifs.gov.au/sites/default/files/fm84d.pdf>. Work undertaken for the Closing The Gap clearing house (Curth-Bibb, J, Moran, M and Porter, D 2014, Funding Indigenous organisations: improving governance performance through innovations in public finance management in remote Australia, <http://www.aihw.gov.au/uploadedFiles/ClosingTheGap/Content/Our_publications/2014/ctgc-ip11.pdf>) found that flexibility was a key component of success for contracts with Indigenous organisations. An evaluation of the Home Care Packages programme and Consumer Directed Care is currently being undertaken by KPMG, with results to be published in 2015. Further information is available from Department of Social Services 2015, 'Evaluation of the Home Care packages Programme and Consumer Directed Care', <https://www.dss.gov.au/our-responsibilities/ageing-and-aged-care/aged-care-reform/reforms-by-topic/home-care/evaluation-of-the-home-care-packages-programme-and-consumer-directed-care>. The Digital Transformation Office outline undertaking an experimental approach as an important principle: Digital Transformation Office 2015, 'Principles of digital service', <https://www.dto.gov.au/standard/principles-digital-service-design>
 See for example the Australian Centre for Social Innovation's Family by Family program in South Australia and NSW, The Australian Centre for Social Innovation 2015, Family by Family program, <http://tacsi.org.au/project/family-by-family/>; and Uniting Care Burnside's 'Newpin bond' in NSW: NSW Department of Premier and Cabinet 2015, 'Social benefit bonds', <http://www.dpc.nsw.gov.au/programs_and_services/social_impact_investment/social_benefit_bonds#nsw_social_benefit_bond_trial>
 See Department of Finance 2015, 'Managing performance', < http://www.finance.gov.au/resource-management/performance/>
 NSW Department of Premier and Cabinet 2015, 'Social benefit bonds'.
 Joint Standing Committee on Foreign Affairs, Defence and Trade 2015, 'Partnering for the greater good—the role of the private sector in promoting economic growth and reducing poverty in the Indo-Pacific region,' Inquiry of the Foreign Affairs and Aid Sub-Committee, Recommendation 22, <http://www.aph.gov.au/Parliamentary_Business/Committees/Joint/Foreign_Affairs_Defence_and_Trade/Indo-Pacfic_Economic_Growth/Report>
 Sundheim, D 2013, 'To increase innovation, take the sting out of failure', Harvard Business Review, January 9 2013, <https://hbr.org/2013/01/to-increase-innovation-take-th>
 The Behavioural Insights Team 2014-2015, 'What are behavioural insights?', <http://www.behaviouralinsights.co.uk/about-us>
 The White House Social and Behavioral Science Team is part of the Office of Science and Technology Policy. See White House Social and Behavioral Science Team n.d, 'Using behavioural science insights to make Government more effective, simpler and more people-friendly', <https://www.whitehouse.gov/blog/2015/02/09/using-behavioral-science-insights-make-government-more-effective-simpler-and-more-us >
 Hawke, A 2010, p. 29.
 See the Australian Tax Office's project to improve payment compliance, Australian Taxation Office 2014, 'Better communication to improve payment compliance', <https://www.ato.gov.au/About-ATO/Access,-accountability-and-reporting/Informing-the-community/Our-effectiveness/Fostering-willing-participation/Better-communication-to-improve-payment-compliance/>; and the joint report between CSIRO and the Department of Human Services 2014, Transforming human services for the digital era, <https://publications.csiro.au/rpr/download?pid=csiro:EP149489&dsid=DS2>
 Public Governance, Performance and Accountability Act 2013, section 17.
 Parkinson, M 2014, 'A Treasury for the future: organisational change and Treasury's capability', Speech presented at the Institute of Public Administration Australia, Canberra, 20 March, <http://www.treasury.gov.au/PublicationsAndMedia/Speeches/2014/IPAA>
 Department of Social Services 2015, 'Consumer directed care (CDC) in home care packages', <https://www.dss.gov.au/our-responsibilities/ageing-and-aged-care/overview/advice-to-the-aged-care-industry/consumer-directed-care-cdc-in-home-care-packages>
 Department of Finance 2009, National Government Information Sharing Strategy, pp. 12-13, <http://www.finance.gov.au/files/2012/04/ngiss.pdf>.The issue of information silos is discussed in the following report: Office of the Information Commissioner 2013, Open Public Sector Information: From principles to Practice, <http://www.oaic.gov.au/information-policy/information-policy-resources/information-policy-reports/open-public-sector-information-from-principles-to-practice>
 Cutler, T 2008, Venturous Australia – building strength in innovation, (Cutler Report), p. 21,<http://pandora.nla.gov.au/pan/88661/20080911508/www.innovation.gov.au/innovationreview/Documents/NIS-summary-web1.pdf>
 Australian Research Council 2015, 'ARC Open Access Policy', <http://www.arc.gov.au/arc-open-access-policy>
 National Institutes of Health 2004, 'Frequently Asked Questions', <http://grants.nih.gov/grants/policy/data_sharing/data_sharing_faqs.htm#911>
 Examples include The Institute for Government < http://www.instituteforgovernment.org.uk/>; Nesta< http://www.nesta.org.uk/>; and the Alliance for Useful Evidence <http://www.alliance4usefulevidence.org/>.
 Johnson, M 2014, 'Before the Decade is OUT: What if the "Giants of the Web" designed Government Service Delivery', <http://delimiter.com.au/2014/03/24/decade-giants-web-designed-government-service-delivery>
 Sturgess, G 2015, Contestability in public services: an alternative to outsourcing, ANZSOG Research Monograph, Melbourne.
 Breckon, J 2015, Better public services through experimental government, Alliance for Useful Evidence, United Kingdom, p. 5.
 About GovHack, sourced from <https://www.govhack.org/>.
 Government 2.0 Taskforce 2009, Engage—Getting on with Government 2.0, Department of Finance, Canberra, <http://www.finance.gov.au/publications/gov20taskforcereport/doc/Government20TaskforceReport.pdf>
 United Nations Department of Economic and Social Affairs 2014, United Nations E-Government Survey 2014, p. 217, <http://unpan3.un.org/egovkb/en-us/Reports/UN-E-Government-Survey-2014>
 Lateral Economics Pty Ltd 2014, Open for business: how open data can help achieve the G20 Growth Target, commissioned by Omidyar Network, pp. 34 – 35, <https://www.omidyar.com/sites/default/files/file_archive/insights/ON%20Report_061114_FNL.pdf>
 Barnes M, Skelcher C, Beiren H, Dalziel R, Jeffares S & Wilson L 2008, Designing citizen-centred governance, Joseph Rowntree Foundation, University of Birmingham, United Kingdom, <http://www.jrf.org.uk/system/files/2205-governance-citizens-engagement.pdf>
 UK Cabinet Office 2008, UK governance approach to public sector reform, United Kingdom; Martin, S 2006, Promoting effective leadership: citizenship and community empowerment, Office of the Deputy Prime Minister, United Kingdom; Joseph Rowntree Foundation 2009, Citizen involvement in local government, <http://www.jrf.org.uk/sites/files/jrf/citizen-involvement-governance-summary.pdf>
 WA Department of the Premier and Cabinet 2006, 'Working together: involving community and stakeholders in decision-making', Office of Citizens and Civics, Perth, p. 33.
 Whittaker, J 2014, 'How a banker delivered happy customers to the NSW Government', The Mandarin,22 August, < http://www.themandarin.com.au/2078-customer-service-nsw-banker-comes-rescue/>
 Australian Government Web Guide 2012, 'Government 2.0 Tools', <http://webguide.gov.au/web-2-0/gov-2-0-primer/government-2-0-tools/#ideas>
 Australian Government Web Guide 2012, 'Government 2.0 Tools'.
 Department of Finance n.d, 'About 'data.gov.au', <http://data.gov.au/about.>
 Smith, G & Wales, C 2002, 'Citizen's juries and deliberative democracy', Political Studies, Vol. 48, Issue 1, pp. 51-65.
 United Kingdom Department for Business, Innovation and Skills n.d, 'We Asked, You Said, We Did', <https://bisgovuk.citizenspace.com/we_asked_you_said.>
 See the South Australian Government's use of a citizen's jury to agree fair apportionment of infrastructure funding for the South-East drainage network, newDemocracy Foundation 2015, 'Who pays? Agreeing fair shares in infrastructure funding (South Eastern drainage)', <http://www.newdemocracy.com.au/our-work/180-south-australian-minister-for-the-environment>
 See the Western Australia's use of a citizen's jury to develop and implement sustainability plans in the Greater Geraldton City Region, newDemocracy Foundation 2015, 'Geraldton 2029 and beyond', <'http://www.newdemocracy.com.au/library/case-studies/oceania/142-geraldton-2029-and-beyond>
 The first GovHack event was in 2009, funded by the Gov2.0 Taskforce, < https://www.govhack.org/>.
 Easton, S 2015, 'Martin Bowles: we're changing the culture at Health', The Mandarin, 8 July, online<http://www.themandarin.com.au/42988-innovation-month-summit-2015-martin-bowles/>
 William Gibson, quoted in The Economist 2003, 'Books of the Year 2003', 4 December.
 NSW Public Service Commission Advisory Board 2014, Doing things differently: raising productivity, improving service and enhancing collaboration across the NSW Public Sector, p. 5, <http://www.psc.nsw.gov.au/about-the-public-sector/doing-things-differently>
 NSW Public Service Commission Advisory Board 2014, p. 4.
 NSW Public Service Commission 2014, Doing Things Differently: Raising productivity, Improving Service and Enhancing Collaboration across the NSW Public Sector, p. 4.
 Ergas, H 2009, 'Is government the best risk manager?', Policy, Vol. 25, No 3, Spring 2009; Gruen, N 2012, '"Soft" secrecy in the media age', blog post, Lowy Interpreter, 12 March, <http://www.lowyinterpreter.org/post/2012/03/12/Soft-secrecy-in-the-media-age.aspx>
 Shergold, P quoted in P Crisp (ed.), So you want to be a leader: influential people reveal how to succeed in public life, Hybrid Publishers, Melbourne, p. 109; Karvelas, P 2013, 'My 20 years of failure to close the gap', Weekend Australian, 1 June.
 Oliver, Alex 2015, The Lowy Institute Poll 2015, p. 15, <http://www.lowyinstitute.org/lowyinstitutepollinteractive/wp-content/uploads/2015/06/2015-Lowy-Institute-Poll.pdf>