Reflections on failure
Last updated: 05 Feb 2016
This page is: current
Download this publication
- Learning from failure (8.5 MB)
Browse this publication
- Learning from Failure: why large government policy initiatives have gone so badly wrong in the past and how the chances of success in the future can be improved
- Terms of reference
- The future: Avoiding repetition of failure
- Executive summary: 28 proposals for improvement
- Reflections on failure
- Lessons from the past
- A: Providing robust advice
- B: Supporting decision making
- C: Creating a positive risk culture
- D: Enhancing project management
- E: Opening up the APS
- F: Embracing adaptive government
- The future: Learning from mistakes
Peter Shergold AC
"Leadership begins with finding the courage to say, 'I accept personal responsibility for contributing to the failure to which I was a party'. That recognition can steer the resolve to make changes, try again and do better. Acknowledging errors publically is a form of self-improvement, not self-abnegation. Failure, and how we respond to it, is where leadership is born." (2015) 
Peter Shergold was the Secretary of the Department of the Prime Minister and Cabinet from 2002 to 2007
No matter how objective an inquiry intends to be, it necessarily reflects the distinctive perspective that the reviewer brings to the task. This report is no exception. I have been drawn to the task because I am genuinely interested in how government processes for implementing large public programs can be improved. The question is important. I am well aware that the review has been prompted by the manifold failures identified by the Royal Commission into the Home Insulation Program (HIP), undertaken by Ian Hanger AM QC, as well as by the array of problems associated with the design and delivery of the National Broadband Network (NBN). But, as my terms of reference suggest, similar issues have been evident in other national projects. Of course, the challenges are not confined to Australia. The Blunders of Our Governments, published in 2013, catalogues three decades of big, failed projects in the United Kingdom.
With the help and assistance of a dedicated Secretariat team, I have sought to reflect critically on the lessons that need to be learned by public servants about how to execute such programs more effectively in the future. I have arrived at conclusions, informed by discussions that have been held with present and former members of the APS, as well as by suggestions made by those in the business and community sectors who have experience of working closely with governments. I am particularly grateful to the peer reviewers who provided their insightful comments on a draft of this report. Many of my sentiments have been expressed already by politicians and public servants (as is evident from the boxed quotations). Indeed a number of my proposals are already in train. They need to be pursued in a systematic way with greater vigour.
It is important to emphasise, however, that the report reflects my own conclusions. It has been drafted from the viewpoint of someone who has been a senior public servant, who remains deeply engaged with matters of public policy and who believes profoundly that the quality of public administration is of great consequence to the manner in which Australia is governed. It would be foolish not to admit up-front that the judgements at which I arrive have been significantly influenced by my own experience. They are one perspective, but I hope a useful one.
I had the singular good fortune to be a senior public servant in the Australian government for two decades. I enjoyed extraordinary opportunities to contribute, in large and small ways, to matters of great public importance. Of course, even when I exerted positional authority, I was on occasion frustrated in my ambitions by the lumbering scale of administrative hierarchy. The APS, I came to realise, is a brilliant creation, delivering a huge number of transactions every day in an efficient and ethical manner. Yet it remains shaped by its origins as an industrial scale, command-and-control organisation. A century ago it mostly delivered letters: today, mostly welfare payments. Officialdom still weighs heavily on it. Bureaucratic processes, just as much as drawn-out political negotiation, can slow the way in which things are done. Innovation can be stymied. Yet I could always discern the significant purpose of the job, enjoy the chance to make a difference and recognise the need to exercise influence with integrity.
Such attitudes are not limited to the senior ranks of the APS. The ethos of public service runs deep. Indeed, I have been frequently inspired by those more junior public servants, often outside Canberra, who have greater direct contact with citizens and for that very reason find their jobs rewarding and fulfilling. They often display considerable ingenuity in seeking creative ways to improve the service that they offer. Do not imagine that they are paper-pushers, dedicated to circumlocution. Rather, think of frontline workers as talent spotters for new ideas and discerning critics of existing approaches. To a very large extent, they are the face, or voice, of public service to the 'customers' of government.
Public service was not my lifetime career. I had worked for a long period as a university historian before I entered the APS. Since leaving the Service, I have spent eight years developing a portfolio career of non-executive positions in private, public, community and academic governance. But on reflection I realise that public service was my vocation. I slowly came to discern its challenges. To take a silent vow of non-partisanship in order to offer impartial advice and to serve successive governments with equal commitment is not a decision to be taken lightly. It involves sacrifices. For most public servants in senior positions, it can be a tough gig. There is a personal cost that goes with wielding influence from the inside.
Yet, too rarely recognised in educational textbooks, an apolitical public administration is fundamental to the good health of Westminster-style democracy in Australia. First-year university students of political science soon learn that the exercise of executive authority by the government of the day is constrained by the need to wield it through the legislative authority of the Parliament, and that its decisions are subject to the scrutiny of an independent judiciary with its own powers. They are introduced to the complex and ambiguous relationship of Commonwealth, State and Territory jurisdictions operating in a Federal structure, founded on a written Constitution.
Generally, much less attention is devoted to the role of professional public administration in a participatory democracy, and the manner in which it influences power. Perhaps that is because much of what senior public servants do is necessarily hidden from view. The role of appointed departmental Secretaries, their executive teams (and increasingly their middle managers) is to provide advice to the elected ministers they serve. On occasion, to employ a well-worn cliché, that needs to be done in a frank and fearless fashion.
Yet—and here's the rub—however forceful and robust the advice that is conveyed to ministers or their political advisers in private, it is vital that once government casts its judgement, its policies should be implemented with energy and dedication. Cabinet decisions need to be given effect, administratively or through legislation, by public servants. No matter whether the policy proposal has emanated from the APS or the minister's office; no matter whether public service advice has been accepted, adopted, adapted or ignored; no matter whether public servants think that the outcome is brilliant or foolish (or even, perhaps, courageous)—their immediate task is to execute decisions on time, on budget and to the government's expectations.
Equally important, a political Opposition or an enquiring media should not be able to discern from words or actions a senior public servant's views on the wisdom of the policy. A government should be judged by the public for the decisions that it makes, rather than for the decisions it might have made had it been more persuaded by the informed entreaties of its administrators. In short, confidentiality lies at the heart of the democratic processes within which policy is deliberated upon. Without it, government decision making and public policy outcomes will be the poorer—outcomes like the HIP will be more likely.
In contrast, the manner in which ministers and public servants administer policy is appropriately subject to intense scrutiny. At the Commonwealth level, they face rigorous scrutiny before a range of Parliamentary bodies, not least at regular hearings of the Senate Estimates Committees. An independent Auditor-General routinely evaluates departmental performance in delivering programs or overseeing projects and, on occasion, the assessments have been scathing. Individual citizens, on occasion with the support of the Commonwealth Ombudsman, and through the use of Freedom of Information (FOI) laws, are able to gain access to the basis of decisions that directly affect them. The Ombudsman will investigate their complaints to see if those decisions were wrong, unjust, discriminatory or just plain unfair. Citizens can also challenge the decisions to which they have been subject before the Administrative Appeals Tribunal (AAT), an institution which promises to provide prompt review with as little formality and technicality as possible. They can seek redress through the courts.
This is as it should be. Public servants exert considerable power over the public that they serve, and it is important that they are held accountable to the Australian community for their actions, within the framework of ministerial responsibility. Mistakes can be costly, and not just because taxpayers' funds may be wasted, misapplied or used profligately. poor administration can, on occasion, deprive citizens wrongly of their liberties, constrain the application of their rights, fail to inform them of their responsibilities or even, tragically, cost them their lives. Public servants need to answer for their actions. They remain accountable even when, as now, so many government services are delivered under contract by outsourced providers.
Policy sits at the interstice between what is confidential (the development of policy) and what is public (its delivery). For that reason, when things go wrong, as they did so profoundly in the HIP, it is often difficult to attribute responsibility. I have an instinctive sympathy for public servants who find themselves subject to criticism and perhaps disciplinary action for their failures. That is not just because when big but unintentional mistakes occur, one looks at one's own career and thinks quietly, "there, but for the grace of God, go I". It is also because memories of what exactly happened are notoriously unreliable, especially when the written record of decision-making is sketchy. In an environment in which decision-making is too often opaque and responsibility diffuse, it can be difficult to attribute blame. Indeed, the attempt can create a sense of injustice that inhibits a proper understanding of the array of reasons why events went so terribly wrong. public servants are often aggrieved by accounts of their avowed incompetence: often they feel that they have worked around the clock on a large project, had their good advice ignored, done their best to implement the government's decisions and then borne the blame when things went awry.
The hearings held by the Royal Commissioner show clear evidence of this tendency. In general, when projects turn out well, one's mind will naturally tend to ascribe a greater significance to one's own role; and when decisions go badly, it can be comforting to think that one's actions were of relatively minor consequence in a system-wide organisational failure.
Yet it is crucial that organisations and individuals are able to learn from their mistakes. Thankfully the evidence that is available from enquiries into the design and delivery of the HIP, the roll-out of the NBN, the construction of school buildings as part of the Building the Education Revolution (BER) policy—and from a range of other major government programs that have been marked by managerial shortcomings—suggests that there are a number of common factors that reduce the chances of success. There are also measures that can enhance the prospect of positive outcomes. Understanding both threats and opportunities can help to increase the likelihood of effective implementation of major projects in the future.
Over the past few months I have read, discussed and thought about these matters, focusing on the delivery of large programs and the development of the policies that underpinned them. I have come to the conclusion that there are ten key lessons that emerge and that can be learned. These are the matters that this review seeks to address.
First, policy is only as good as the manner in which it is implemented. The development of a policy and its delivery are inextricably linked. Implementation should be integral to policy design. A policy cannot be elegant if its execution is poorly communicated, ineptly administered or inadequately evaluated. The proof of the pudding is in the eating.
Second, policy advice can only be frank and fearless if it is supported by written argument. Good government is founded on good policy, and good policy depends on forthright advice. Ministers look to their public servants (although not to them alone) to provide or challenge ideas. discussion of proposals should be robust. Yet oral advice, however strongly it is conveyed, can too easily be ignored or misinterpreted. Worse, it becomes possible to argue about whether it was actually given or received.
Third, deliberations, oral and in writing, need to be protected. Ministers (and their advisers) and Secretaries (and their senior executives) need to have complete trust that public service advice will remain confidential. If private discussions become public, mutual respect will be eroded. If confidentiality is not assured, public servants will be tempted to temper their counsel and ministers will prefer to receive advice only orally. That's a bad outcome for governance. Anodyne advice undermines effective decision-making. Oral advice leaves no trace of the reasoning behind the decisions that were made.
Fourth, deliberative documents need to be preserved, whether written on paper or delivered by digital means . Nothing symbolises significance more than handing a minister a sheet of signed advice. Increasingly, though, policy is developed in real time by email and text message communication between departments and ministers' offices. These important electronic documents need to be managed as confidential records. They are the files of the future. They are our protection against 'digital amnesia'.
Fifth, it is up to ministers, not officials, to make policy decisions. The important role of senior public servants is to ensure that Cabinet ministers make their decisions with eyes wide open. Advice should seek to identify the risks, envisage unintended consequences, indicate threats to successful implementation and proffer alternative options. Public servants should not seek to impede a government's ambitions but to help it find the best way to give them effect, ensuring that ministers are cognisant of the full ramifications of their decisions and the impact that they will have on business and community interests and on the general public.
Sixth, the effective management of risk is just as important in the public sector as in the private—perhaps more so. Governments need to be apprised of the strategic, financial, operational, regulatory and political risks to the implementation of major projects. This is particularly the case when successful implementation depends on the co-operation of other tiers of government and/or delivery by third-party agents. Public servants need to indicate how risks can be mitigated or accepted, not avoided. Problems often emerge a long way from Canberra. Risk culture and behaviours need to be embedded across the whole department, from the departmental Secretary to the most junior regional officer. Prevention is the first line of defence.
Seventh, as the public service fully commits itself to measuring results by outcomes, program management needs to be accorded far greater professional status. The Public Service Act 1999 places equal emphasis on members of the Senior Executive Services (SES) exhibiting the capacity to provide policy advice or deliver programs. That is not reflected in cultural attitudes or behaviours. Too often in the APS, policy advice is accorded significantly greater status than program delivery. In truth, conceptual, analytical and strategic skills (the foundation of SES selection) are just as important to program management as they are to policy design. Project and program managers need to be recognised as a community of practice in the public service, with attributes as important as those who possess legal, accounting, medical or veterinary skills. Capability needs to be progressively enhanced across the APS in a systematic fashion. In the past, public servants tended to administer by the prescription of process. Management directed its attention to the distribution of allocated inputs. Today the focus is far more heavily on performance-based outcomes. Results matter. New professional skills are required.
Eighth, good governance increasingly depends on collaboration across sectors. The private and community sectors are now central to the construction of public infrastructure and delivery of public programs. They are commissioned to deliver government services. The public market is becoming contestable, offering citizens greater choice. Unfortunately, too often the relationship with providers is conceived merely in contractual terms. Their expertise is not given full expression. Public servants need to facilitate the engagement of business leaders, not-for-profit CEOs and senior academics in the design of major projects. They should be seen as partners rather than as stakeholders. Their experience should be tapped. That will help to ensure that there is more evidence-based policy and less policy-based evidence. The key to transforming the delivery of major projects is to focus less on contract compliance and more on collaborative performance.
Ninth, the APS needs to be further opened up. More opportunities should be found for those who work in business and community organisations to work in the public arena. This is not to suggest that public servants are second class. An increasingly graduate workforce is probably more qualified and capable than ever before. However, there is considerable benefit in increasing the diversity of perspectives brought to public administration. Conversely, public servants should be actively encouraged to undertake periods of relevant work in companies, social enterprises or universities. Greater experience will enhance their performance when they return to the APS. There is significant advantage in enabling people to stand in the shoes of others, appreciating the different constraints under which they operate, and learning new approaches. Such exchanges will not undermine the public sector ethos but reinvigorate it.
Tenth, an adaptive government can respond rapidly to changing circumstances without taking unnecessary (and unforeseen) risks. On occasion, governments have an appropriate desire to act quickly. Public servants are often seen as an impediment. They can be perceived as cautious, guarded, even unimaginative. They can seem risk-averse. Yet their circumspection is based on the knowledge that the rollout of major national programs is fraught with danger. With wicked, complex and deep-seated public problems, it is uncertain exactly what policies will work, or how they should be delivered in the most effective way. There are benefits to experimentation: often it is more sensible to test out ideas on a small scale rather than across the whole nation. More attention should be given to using trial or demonstration sites to begin implementation expeditiously, trialling different delivery options and learning by doing. Success can be demonstrated early. Failure can be addressed fast.
These views frame the report that follows. It endeavours to do justice to the perspicacious insights of the Royal Commissioner, Mr Ian Hanger AM QC, and others who have reviewed the mistakes made by those who have designed and executed major government programs. I have sought to listen empathically to the responses of public servants to these swingeing criticisms. I have tried not to cast judgement on individuals. I admire and respect the APS. While I have not shied away from frankness in acknowledging its past failures, I hope that the conclusions at which I have arrived will help the process of improving the manner in which the APS wields its significant power in the future.
Several of my conclusions reflect a need for the APS to critically examine and reshape how it does its business. In places I have recommended new structures, not for their own sake but as means to an end. New tools, administrative structures and bureaucratic positions can too easily become red tape. Anything that becomes routinised in government process can invoke a compliance mindset. Some of my suggestions will sit awkwardly with established culture and practices. They point to the need for changes that some will find uncomfortable, but which need to be pursued. These are not superficial changes. They will be demanding and even at times troubling. The goal is to build a more mature culture, a more constructive environment and a more sophisticated public sector approach. The end result, while it will take time, will be worth it.
This report, whilst transmitted to the Australian Government, is intended to open up 'strategic conversations' across public services. I hope that its arguments will inform public discourse, and perhaps even stimulate heated debate. If so, my review will in large measure have achieved its objectives.
 Shergold P 2015, 'Mea culpa—failure as the foundation of leadership,' in P Crisp (ed.), So you want to be a leader: influential people reveal how to succeed in public life, Hybrid Publishers, Melbourne, p. 109.
 King, A & Crew, I 2013, The Blunders of our governments, Oneworld publications, London.