Editors’ Note: This article is one of two companion papers published simultaneously in this Journal. Both papers examine a CDC initiative on public engagement and public policy decision making.

Abstract

Summary:
Between 2005 and 2009, a new model for decision-focused public engagement, now called a “Decision-focused Public Engagement Table” (DPET), was developed and nine public engagement projects were carried out with this model or modified versions of it. The model is distinctive for its inclusion in a decision making process of both citizens-at-large who have no special interests as well as stakeholder-citizens representing organizations in the key sectors affected by the decision. Each project addressed a difficult-to-resolve, values-laden science policy question related to pandemic influenza or other public health topic.

These projects were implemented under the sponsorship of the Centers for Disease Control and Prevention (CDC), the Department of Health and Human Services (HHS), or the Association of State and Territorial Health Officials (ASTHO). They were guided by a set of principles culled from best practices in the field and well-aligned with the 2008 broadly-applicable findings on public participation by the National Research Council of the National Academies.

Taken as a whole, the projects provided further “proof of principle” that engaging the public in participatory policy-making sponsored by governmental agencies is not only possible but can be influential in shaping decisions. A companion paper in this Journal provides detailed case abstracts for each project and an overall summary and discussion. [1] This report describes ten lessons learned from conducting these projects.

Each of the nine projects took place in a different context. The first lesson learned is that agency support for public engagement is often absent or weak. Also, agency officials have widely varying concepts of what is involved in engaging the public.

From a project management perspective, the lesson learned is that no public engagement project is ever perfect, and that determining when to go ahead or cancel an imperfect project can be a difficult but necessary decision. Also, obtaining clarity about the purpose of the project, the decision to be made, and the decision making process are important and difficult tasks confronting organizers.

From implementation of the projects, the lesson learned is that not all types of work shared with participants within a decision making process are equally demanding, equally satisfying, or equally rewarding for them. We also learned that public engagement appears deceptively simple to carry out, but is difficult to implement well. And we learned that the general public weighs choices differently than professionals and stakeholders who view the world from a narrower lens.

From an evaluation perspective, we learned that public engagement is difficult to evaluate accurately because of multiple sources of advice. Also, evaluating public engagement in an unbiased fashion is challenging in a democratic society where everyone feels obliged to say positive things about democratic practices even though they may believe differently.

From an organizational perspective, the lesson learned is that reaping the full benefits of engaging the public will require creating an organizational “home” within the agency or sponsor. A final lesson was that collaboration with organizations outside of government helps (and may even be necessary) to implement projects effectively and expeditiously.

Several potential approaches are suggested to overcome the challenges, avoid the pitfalls, and reap the potential benefits in using the DPET model. If implemented, these approaches could help to enhance efforts already underway in the deliberative community to make greater and more effective use of participatory approaches to public policy making. Because they are difficult to resolve, values-laden public policy choices are often stalemated or gridlocked. Policy decisions delayed are policy benefits denied. Decision-focused public engagement can be used as collaborative problem-solving to work through difficult, values-based policy issues. Doing so can produce sounder public policy decisions, and it has the potential to empower participants and build needed public trust in government.

Keywords: Public engagement, decision-focused public engagement table (DPET), pandemic influenza planning.
Citation: Bernier RH. Lessons learned from implementing a multi-year, multi-project public engagement initiative to better inform governmental public health policy decisions. J Participat Med. 2014 May 22; 6:e8.
Published: May 22, 2014.
Disclosures: The author has declared that no competing interests exist. The analysis and opinions expressed here are solely those of the author and not the official views of the CDC or any other government or non-governmental organization. The author was employed at the CDC during the period of time the projects were carried out. The information used for this review was obtained from existing project reports available publicly online, and from published and unpublished manuscripts. Acknowledgments and citations have been provided where appropriate in this manuscript and in a companion paper containing more details about the individual projects published in this issue of the Journal. [1]

Introduction

Prompted by the controversy over vaccines and autism and the concern it reflected by some segments of the US population,[2][3] a new model for actively engaging both citizens-at-large and representatives of stakeholder organizations was developed. The purpose of the new model, now called a “Decision-focused Public Engagement Table” (DPET), was to better inform difficult, values-laden vaccine policy decisions, potentially earn support for them, and build a better, more trusting relationship between CDC and the public.

Beginning in 2005, the DPET model or modified versions of it have been used in nine public engagement projects that have been initiated by CDC and/or the National Vaccine Program Office at the Department of Health and Human Services, the parent Department for the CDC. One of these nine projects was a grant program to six state health departments to assess the feasibility of the DPET model at the local level. A brief description of the nine projects is included in Table 1 of this paper and case abstracts for each project are presented in a companion paper in the Journal.[1] In addition, a third paper describing a proposed participatory evaluation framework by the researchers who evaluated the state and local DPET projects has also been recently published in this Journal. [4] The purpose of this paper is to provide an overview of all the projects and to identify key lessons learned that might provide useful guidance for researchers and practitioners seeking to improve the quality of public engagement projects.

Background

All public engagement projects take place within a particular context or situation with its own unique opportunities and challenges. Each practitioner takes these factors into account when conducting such projects. Also, practitioners of public engagement processes must take into account best practices or guiding principles of sound public engagement when carrying out any particular project. Before presenting the lessons learned from these nine projects, the broader context in which they took place and the principles which were developed to guide them are described. Armed with this knowledge, researchers and practitioners will be more able to determine which lessons learned from these experiences offer them the most useful guidance for the future.

Broad Contextual Factors

Origin: These projects were initiated “top-down” by government agencies or a non-governmental organization (NGO) seeking consultation from the public on pending decisions rather than at the grassroots level by citizens interested in solving a problem or stimulating government to act on a problem. The subject area was chosen by the government, not identified by the public.

Purpose: According to the National Coalition for Dialogue and Deliberation (NCDD), there are four major purposes for using the techniques that are at the heart of public engagement.[5] They are (1) to learn more about people or issues, (2) to resolve conflicts, (3) to influence public decisions, and (4) to act collaboratively to solve problems. Using the DPET model, the nine projects in this report were decision-focused, that is, conducted for the purpose of helping to better inform and potentially influence decision makers on public policy issues. Secondary purposes were to increase support for the final decisions made and to enhance trust between citizens and government.

Problem Solving Stage: Problem solving includes: (1) problem identification or definition, (2) gathering information about or determining the causes of the problem, (3) designing or finding options to address the problem, (4) discussing pros and cons of the options, (5) weighing the options for a solution, (6) choosing a best solution, (7) implementing the chosen solution, and (8) evaluating an implemented solution.

In these decision-focused public engagement projects, the focus was on obtaining the public’s contributions to the discussing, weighing, and choosing stages. These are closer in sequence to the final decision. The implications were that participants would have potentially more direct and traceable influence over the final choice made by decision makers. This type of work for the participants also carried implications for the level of satisfaction they might experience, and for the accountability decision makers would have.

Choice of Publics: Early on in any consultation exercise, sponsors need to make a decision about what publics they wish to engage. The most fundamental choice is whether to include citizens-at-large with no recognized interests or stakeholders who are representatives of organizations with recognized interests. The choice made with the DPET model sought to include both the general public and the stakeholder public. This choice made it more difficult to achieve agreement on a single public perspective to be conveyed to the sponsoring organization, but could have the advantage of presenting more fully considered advice.

Mandates: The agencies implementing this model were not mandated by law to involve citizens in their decision making process. All of the topics placed on the table for public engagement were chosen voluntarily by the sponsors. This feature carried implications for the degree of commitment by the sponsors to consider the advice given by the public.

Governance Level: Also, since the CDC is a federal agency and the Association of State and Territorial Health Officials (ASTHO) represents all state health departments in the US, these projects were carried out, with one exception, as national level projects with the goal of assisting in the work of a federal agency or non-governmental organization with nationwide responsibilities. Such projects have unique elements not found to the same degree at the community level such as challenges in motivating citizens to participate or in achieving adequate representation.

Commitment Level: Finally, the DPET model was initially proposed for ongoing public engagement with creation of a standing agency infrastructure. In practice, these projects were carried out on an ad hoc basis project by project. Creating a standing support infrastructure has implications for the kind and quality of resources and expertise that would be available for individual projects and the amount of training required.

The Guiding Principles

Guiding principles of good public engagement or best practices have been identified over the years by practitioners and researchers in the field. These principles were collected from a variety of publications, literature reviews, and summaries of case studies. Based on the frequency of mention, on discussions with experienced practitioners in the field, and taking into account the context in which the projects were operating, several principles were identified and judged to be important. The principles compiled were consistent with those of the Organization for Economic Cooperation and Development which examined this topic [6] and with the findings and recommendations of the National Research Council of the National Academies. [7] These principles with annotations are included with this paper as Appendix A.

Working Definition

The definition of decision-focused public engagement used to guide each of these projects was “The practice by which the agency or sponsoring organization very actively involves members of the public-at-large and/or representatives of stakeholder organizations in group dialogue and deliberation sessions to better inform and possibly shape the agency or organization’s final policy decisions.”

The Decision-focused Public Engagement Table (DPET) Model

With the working definition of decision-focused public engagement described above, the DPET model had the following key features:

1. A focus on undecided policy choices
2. Inclusion of representative groups of both stakeholders and citizens-at-large
3. Linkage with government decision makers
4. Neutral facilitation
5. Presentation of balanced information from credible sources
6. Frank dialogue and genuine deliberation
7. Reaching some form of group judgment or agreement
8. Synthesis of the results as the “public perspective” on the pending decision
9. Feedback to the participants

The operational characteristics or “nuts and bolts” of the DPET model, such as the number of participants, the length and frequency of the meetings, etc, are presented in Appendix B to this paper. Not all projects had all features since modifications were often made necessary by the timing and resources available.

A brief synopsis of the nine projects covered by the lessons learned is presented in the table below. Actual case abstracts with more details for each project as well as a summary and conclusion of the projects are presented in the companion paper.[1]

Table 1. Decision-focused Public Engagement Projects.

Lessons Learned

Lesson #1:

Support for public engagement among many agency officials is absent or weak.

Contrary to what is often assumed, support for public engagement is not a given. Even when present, it is often limited or weak. [8][9] This reality had repercussions on the numerous activities to be carried out in our projects. For example, support for the work needed to recruit adequate numbers of participants or representative samples sometimes seemed halfhearted. This also seemed true for providing the desired clarity about the goals and purposes of the events, for allotting adequate time for dialogue and deliberation at the meetings, or for giving serious consideration to the results obtained. Therefore, as organizers of these events, we often felt we were swimming against the tide. We thought of four possible reasons for this weak support. First, government officials are accustomed to obtaining public input through polls, surveys, focus groups, written public comments, consumer representatives, and during public comment periods at the end of expert advisory committee meetings. These routine forms of obtaining public input often require minimal effort from public officials and provide for only limited interaction and mutual learning by citizens and public officials.[10] These forms of public input have become the norm and expectation and so the new DPET method represented a significant change.

Second, many public servants have had largely negative experiences in dealing with an angry public in meetings, in processing hostile freedom of information requests, and other adversarial exchanges. Understandably, many agency officials have come to view the public as something that should be kept at arms-length rather than as a potential resource helping to produce better decisions.[11]

Third, public engagement is messy because of conflicting views on the part of citizens and can be difficult to manage. Also, leaders can potentially end up with public judgments which are at variance with government preferences, at least initially. In these situations, agency officials underestimate or ignore the role of competing public values and the qualifications of citizens to participate in fully making tradeoffs and difficult public policy choices.[12][13][14]

Fourth, the US has a representative form of government with elected or appointed officials who not only have the authority but the responsibility to make decisions on behalf of the people. Often these officials have special training and expertise required for the positions they hold, and they may feel obliged to make the decisions. They are under no obligation to consult constituents or citizens before making public policy decisions.

The challenge may go even deeper. Despite the widely held belief that the United States government is a government “of the people, by the people, and for the people”, our society sometimes appears ignorant and skeptical about the contributions the public can make.[12] Though we may refer to “the wisdom of the American people”, we make no concerted effort to uncover and develop that wisdom.[8] We believe the public is the best guardian of the common good, but there are no established institutions of government currently in place to routinely obtain meaningful input from citizens in public policy decision making. Occasionally, during a crisis such as gun violence in schools, the President will recognize “the need to have an honest national conversation”, but these can sometimes be seen as attempts to persuade the public, as much as to listen.[15] With the exception of voting, no active role for citizens has been created and carved out of the everyday process of governing. Lack of public participation is the rule, not the exception.

Also, public engagement is a form power sharing for public officials, and this is often perceived as giving up control. Persons with power and control are not usually inclined to cede it.[16][17] This is exemplified by the issue of gun control in the US. While polls have indicated widespread and even overwhelming public support for additional measures to curb access to guns, the public’s values are not being respected by elected representatives. This is an example of “public values failure.” [18] In short, there is public will but no accompanying political will.

These features about the structure and operation of our government act to hinder the development of a fuller understanding of the potential value of active citizen engagement and adversely affect the willingness of officials to engage meaningfully with citizens, and of citizens to engage confidently with government.

Possible Approaches Stemming From Lesson #1

As the saying goes, “nothing breeds success like success.” [19] While not all of our projects succeeded, the public’s advice in several of our decision-oriented public engagement projects, including the early ones, influenced the content of the final agency level recommendations or policies that were issued. This is the final test of public engagement—that it can produce technically sound decisions which are better as a result of the public input. The early success was publicized and we obtained additional requests for public engagement projects as a result.[20] Thus, demonstrating successful projects has the potential to encourage greater use within agencies and increase the needed support for public engagement.

Thus, it makes sense for public agencies to carry out more high-quality, high-visibility public engagement demonstration projects and evaluations and frame them as continuous quality improvement. The goal would be to examine how to engage citizens in the public policy process and help reduce doubt and skepticism about the value of citizen participation. When these demonstration projects are carried out, it will help if there is widespread communication about their results to help improve the public’s understanding of the potential benefits and pitfalls of well-designed and well-executed projects.

Convincing public officials that public engagement is a smart thing to do, and finding committed public officials willing to conduct these demonstration projects, are challenging for the reasons cited above. Ultimately, the most compelling rationale was articulated centuries ago by an ancient Buddhist prince in 604 AD, “…when big things are at stake, the danger of error is great. Therefore, many should discuss and clarify the matter together so the correct way may be found.” [21]

The long term effects of repeatedly successful and highly publicized model demonstration projects could be to help change the culture from one in which there are deep and widespread doubts about the role of citizens to one where there are new convictions that our citizenry and our participatory mechanisms are useful and essential tools for wiser problem-solving at the society level. [22]

Lesson #2:

Agency officials have widely varying concepts of what it means to engage the public, and sometimes use terms interchangeably to describe similar activities.

Many different individual and group process methods can be used to obtain public input.[23] The methods vary, including the type and amount of information shared, degree of interaction between participants, and the amount of decision making allowed by participants. Some approaches use very little exchange between sponsors and participants have had so little potential for input and influence in the decision making process that they merely pay lip service to engagement.[24][25]

The most frequently used methods at the federal agency level, including submission of written comments to the federal register, answering survey questions, participating in focus groups, serving as a consumer representative on an expert advisory committee, making oral comments during the allotted time at the end of open meetings of advisory committees, or attending town hall type public meetings often fall short of the active participation which we sought to achieve in our decision-focused public engagement projects. [10]

The rationale for undertaking in-depth consultations such as those called for in the DPET model is that anything less than very active interaction or less than a genuine opportunity to impact decision making is unlikely to achieve the benefits of decision-focused public engagement. These objectives are 1) to give people meaningful influence over decisions which affect their lives, 2) make better public policy decisions, 3) earn public support, and 4) build more social capital and trust in government.[26] Achieving these substantive benefits requires genuine commitment to obtaining more active citizen contributions to public policymaking.

Unfortunately, many different terms such as public participation, public involvement, public deliberation, public consultation, public input, public outreach, and public engagement are used interchangeably to describe interacting with the public for the purposes noted above. The absence of a common, standardized definition and an agreed-upon terminology hamper effective communication about public engagement projects, create disagreement about the best design, create false expectations, and make meaningful evaluation more difficult. These flaws and limitations, if not addressed at the beginning of the public engagement process, can produce frustrations and cynicism about government. They make public engagement inadvisable to carry out because they can actually reduce, rather than increase, social capital and lessen capacity to work together effectively to address public problems.[22]

Possible Approaches Stemming From Lesson #2

There are significant differences in the thought and interaction required to contribute meaningful feedback to a sponsor’s ideas, provide new ideas, articulate different viewpoints, exchange ideas respectfully, weigh and make tradeoffs between competing choices, and/or reach group agreement about a preferred course of action.[27]

A blue-ribbon panel of decision-oriented public engagement specialists, government officials, citizens, and stakeholders should be convened to specify the minimum standards of work and interaction for participants, and the advice necessary to achieve the most important benefits from public engagement. The term “decision-focused public engagement” should be defined using these minimum standards, disseminated widely, and reserved for use in situations that meet these standards.

The enhanced levels of work, interaction, and consideration in decision-oriented public engagement may require interested agencies or sponsors to stretch beyond what they are currently doing, but it is realistic to achieve. In circumstances where the engagement project cannot meet the standards, the nature of the contribution being sought after should be described accurately and honestly using a different term or terms.

Lesson #3:

No public engagement project is ever perfect; determining whether to proceed, or cancel an imperfect project, is a difficult but necessary judgment to make.

The Goals Project demonstrated very clearly the tradeoff that can arise between the desire to reap the benefits of public engagement activities and the need to prevent failures which increase public cynicism about government. In that project, a decision was made to move forward despite incomplete preparatory materials, excessive workload for the participants, recruiting of non-representative participants, lack of clarity about the final product, and other warning signs that the project may not succeed.

A frequently heard refrain in the public engagement field is that no single participatory approach or method is right for every situation. And not every situation in which sponsors call for decision-focused public engagement is suitable for this type of public engagement. Suitable situations include those in which sponsors share democratic principles, are genuinely committed to learn from citizens, and will give their advice serious consideration. Likewise, the right models must be used. Choosing a different method which promises less and demands less of participants and lowers the degree of participation (eg, focus groups, public hearings) may constitute a legitimate form of public interaction. However, such low intensity interaction is not the same as true engagement. Some sponsors may want to claim that the public is being engaged on a pending decision and may want to proceed anyway when it would be best to postpone or cancel the project. Public engagement specialists or their organizations may find it difficult to remove themselves from such projects. They may have a conflict of interest between going ahead to earn consultant or contract fees and meeting tough professional standards for decision-focused public engagement. Hoping that circumstances will change during the project, or deciding that ultimate responsibility for the project and use of the information ultimately resides with the sponsor, ignores the welfare of the public participants, and ultimately threatens to decrease social capital.

Possible Approaches Stemming From Lesson # 3

One safeguard practitioners can use is for sponsors to submit to a “readiness test” prior to undertaking any decision-focused public engagement project. The readiness assessment will help to avoid the most common errors leading to failure of decision-focused public engagement. This test consists of a short series of self-directed questions designed 1) to probe the extent of the sponsor’s commitment to democratic values and principles, 2) competencies to conduct the project activities, and 3) willingness to provide the necessary time, staff, and resources for a quality process to take place.

The sponsor might consider providing a memorandum of agreement certifying that the requirements for a genuine public engagement exercise are in place and committing to carry out the project successfully. In some citizen jury projects, contracts have been made requiring the sponsor to explain how it has responded to the recommendations.[27] This may appear to be unduly burdensome on sponsors; however, public values failures or failures to give serious consideration to the advice received from citizens are a frequent outcome, so this approach may help prevent that outcome. The increase in cynicism and the decrease in public trust which accompany unsuccessful public engagement projects justify the added effort required of sponsors and organizers to go the extra mile to demonstrate their level of commitment.

Lesson #4:

Obtaining clarity about the purpose of the public engagement project, the decision(s) to be made, and the decision-making process to be used are the most difficult tasks of the entire project.

Articulating the purpose of a project, specifying the decision to be made, and describing the decision making process to be followed are entirely under the control of the sponsor and organizers. Thus, they would appear to be the earliest and easiest tasks to carry out. They are the most important to accomplish, yet they can prove to be the most difficult.

Although there does not appear to be a conviction about the value of involving the public in decision making,, a feeling of obligation to consider the public’s viewpoint on key policy decisions is widespread among agency officials.[10] This may sometimes lead to ‘going through the motions’ of obtaining public input or giving the appearance of having done so without intending to give it serious consideration. Coupled with the fact that public engagement is poorly understood and the terms are ambiguous, this tendency to pay lip service to public engagement only makes obtaining clarity about the true purpose of a public consultation process more difficult to achieve.

Sometimes it seems that a group process for the sponsors before the group process for the public would be helpful in order to reach the kind of clarity of purpose and decision necessary to guide execution of a public engagement project. Since support by sponsors for “a process before the process” is often impractical, project staff must strive to reduce ambiguity and achieve clarity in order to succeed.

Possible Approaches Stemming From Lesson # 4

It can be helpful in seeking clarity to ask decision makers to specify the decision-related product they expect to create (eg, a recommendation, a set of options), and to think about how and when this product will be created through this process.

For example, in some projects, the final product desired was a recommendation or preferred choice from among several options presented. In the projects on vaccine prioritization, the desired product was a set of recommendations about who should be vaccinated first, a clearly discernible output. In the project on At Risk Populations, the decision makers believed they had gaps in their understanding of the issue and in the definition of at-risk populations, and they set out specifically to close those gaps with information from the at-risk populations to be served by the policy decisions. This also resulted in a very useful outcome. In contrast, in the Goals project, the agency misjudged the product it wanted since it never created a final list of priority objectives. Instead, there was never a chance for the public input to prove useful or influential in shaping the new final product. Due diligence is required before moving ahead with decision-focused public engagement projects because failure can cause harm and wastes resources. [28]

Lesson #5 :

Not all types of work called for at different stages in the decision making process are equally demanding of participants and therefore not all are equally rewarding.

The decision making process can be categorized as having early, upstream elements related to gathering information and setting up the decision, and downstream elements that take place closer to the decision. Examples of such upstream activities are problem detection, framing the problem, and identification of the causes of the problem. Examples of downstream activities are generation of options or potential solutions, identifying and discussing the pros and cons of each option or solution, weighing the pros and cons of each with the intended as well as unintended consequences, and seeking consensus or reaching a course of action or making a recommendation that the participants agree with.
Some projects will involve participants primarily or exclusively in the early formative stages of the decision making process when the decision is still taking shape. Brainstorming ideas and feedback obtained at these early stages can prove useful to decision makers later on in the process when the decision is being made. However, such preliminary input is often still vague, hard to keep track of, and easy to ignore or consider as no longer relevant in the final stage of the decision making process. This contrasts with the more specific and clear advice that can emerge from recommendations made or preferences expressed in the later stages of the decision making process. Also, the earlier, upstream input makes holding decision makers accountable more difficult. Some decision makers may claim the public was consulted and the advice was useful without it being possible for the public to trace or verify the accuracy of these claims.

Possible Approaches Stemming From Lesson # 5

The early stage work associated with developing a policy decision should not be identified as decision-oriented public engagement but by some other term such as public brainstorming, ideas generating, or public framing. It consists mainly of giving ideas or input. This can be useful, but is a limited role for participants. Sponsors and organizers should seek to engage citizens in later stage “decision-ready” work which requires a very active engagement to bring about a candid exchange of views and mutual learning. Such a role can be more satisfying and empowering for participants, is more demanding of them and is more directly and tangibly related to the decision made. [29]

Lesson #6 :

Public engagement appears deceptively simple to carry out, but is actually very difficult to implement well.

Decision-focused public engagement can be understood as bringing together the right people to learn from each other through discussion in an effort to reach agreement about a decision or future course of action. Bringing people together to work on a topic which is not of prime interest to them is very difficult without creating new incentives. Bringing together a group of persons which is representative of the population is not easy. Bringing stakeholders together who have busy schedules and may have drastically different views about the pending decision on the table is never easy. Once assembled, structuring conversations between a large group of participants, many of whom are strangers to each other, requires skill in using techniques that make people feel comfortable when exchanging views.

Furthermore, public engagement for decision making requires allowing adequate time for airing different viewpoints, hearing pros and cons, weighing the consequences of different options, and making a choice about which option is most responsive to participants’ values. While it may be appear easy in some situations to carry out some of these elements, others are always very challenging with large and diverse groups, and it is extremely difficult to carry out all of them in a high quality fashion. And the ultimate credibility of the results of a public engagement process can often hinge on the capacity to execute all of these elements in a high quality fashion.

Also, participatory approaches can be “messy” and difficult to control. These characteristics of democratic practices require a “faith in the process” which is sometimes difficult to muster. For one thing, decision makers have witnessed or participated in many public processes which have been disappointing. The town hall meetings conducted on health reform in the summer of 2009 are often cited as an example of failed and discouraging public meetings. [30] Also, when agencies have a specific pending decision to make, and when investments in public engagement have been large, it can be disconcerting not to be able to guarantee a useful product at the end of the process.

Possible Approaches Stemming From Lesson #6

Practitioners and sponsors must face up to the fact that quality public engagement processes are not easy to implement and they must take all of the necessary precautions to avoid pitfalls and to put safeguards into place. For example, creating and enforcing ground rules for how the participants will conduct themselves and make decisions as a group are critical and were widely applied in these projects, particularly the one on vaccine safety. Having more widely adopted standards of good practice about decision-oriented public engagement and having specific elements and best practices for a particular type of public interaction will greatly assist future efforts. Making the efforts to conduct quality public engagement processes will help assure a worthwhile return on investments with more sound, more supportable decisions, more empowered participants, and more social capital. It will also help to convince leaders who do not currently support public engagement of its value.

Lesson #7
The general public weighs decisions differently than professionals and stakeholders (who view the world from a narrower lens).

In the first pandemic influenza vaccine prioritization project both the citizens-at-large and stakeholder organizations decided with a very high level of agreement that assuring the functioning of society should be the first immunization goal in a moderately severe pandemic, followed in importance by reducing individual deaths and hospitalizations. There was little support for other suggested goals, ie, to vaccinate young people first, to use a lottery system, or to take a first come, first served approach.

When the first prioritization project was developed, the plan was for the citizen and stakeholder work to inform the decision-making of the two expert committees that routinely advise HHS on vaccine issues. Unfortunately, these expert committees had already met jointly before the public engagement work got off the ground.

The expert committees and the public reached quite different conclusions about who should be vaccinated first: the experts prioritized decreasing health impacts by vaccinating people at high risk of death and hospitalization, while the public prioritized minimizing societal impacts by vaccinating people tasked with key social functions (eg, fire, police, EMS).

In issuing its first set of national guidance in November 2005, HHS accepted the advice of its expert committees.[31] While the results of the first public engagement project did not directly shape the content of the first national guidance, they did compel HHS to question the finality of its expert-derived recommendations and to see the public’s viewpoint as the basis for further discussion and exploration. And, approximately one year after the initial guidance was issued, HHS decided to revisit it.

In the fall of 2006, a new interagency workgroup was established under the leadership of HHS and the Department of Homeland Security (DHS) to set forth new guidance. The interagency group conducted a second public engagement project in 2007 using the DPET model which came to conclusions similar to those of the first public engagement project. This time, however, leadership adopted many of the public’s recommendations in preparing its national guidance issued in July, 2008. [32]

The public engagement projects on priorities for the use of pandemic influenza vaccine taught very clearly that expert professionals and stakeholders can reach different conclusions than laypersons when considering the same values-based question. This is perhaps the most significant lesson because it establishes proof of the value of the judgment of citizens-at-large in policy making. It illustrates that the public wisdom can be obtained from a properly constituted public meeting in structured ways. [8][12]

The reasons for this discrepancy of views may have to do with the interests which the expert professionals and stakeholders are seeking to pursue which are narrower than the wider public interest which can be the main focus of citizens-at-large. Also, it may be attributable to the different training of expert professionals and stakeholders which cause them to have a sense of obligation or professional duty to promote a specific course of action which is best aligned with the goals of their profession or interest group.

In this example, health professionals appeared to be viewing the decision primarily from a health perspective — saving lives and preventing hospitalizations — as might be expected of them. However, when health considerations are not the only ones considered when arriving at a decision, a potential problem arises if the health professionals are the primary or only persons consulted.

The discrepancy which occurred between the views of expert stakeholders and the public in weighing the best approach to preparing for a national mass vaccination program against H1N1 in 2009 is yet another example of an expert/public disconnect. In that project, 57% of stakeholders favored an all-out approach to prepare for a mass vaccination program while only 25% of citizens-at-large favored this option. In the end, the low public demand for vaccination which emerged provided evidence that an all-out preparation effort was not the wisest course of action to pursue as the earlier public judgment had concluded. An important lesson learned is that the public non-stakeholder perspective is more likely to represent the broader public interest and not any particular special interest. Both perspectives are sought and have the potential to be reconciled in the DPET model.

Possible Approaches Stemming From Lesson # 7

Several of the methods used for public engagement include only stakeholders or only citizens-at-large. In order to reach the soundest possible public policy decisions, obtaining advice from the sectors of society which may be more impartial and more aligned with the common good is desirable. Likewise, it is important to include the stakeholder-public in order to obtain the best thinking of the most affected sectors of society. These people are likely to have a more detailed understanding of the issue and to have thought more about the consequences of different options, albeit for specific groups.

Thus, representatives of both the public-at-large and the stakeholder-public should be recruited for decision-oriented public engagement projects on pending public policy decisions.[12] Both of these groups were represented in the DPET model and can lead to more fully considered decisions.

Lesson# 8:

Public engagement is difficult to evaluate accurately and in an unbiased manner.

Despite the importance of determining to what extent the advice of the public was considered or heeded in the final decision making, government policy decision making is often based on input from a wide variety of experts and interest groups. It can be difficult or impossible to disaggregate the component elements of a decision that were contributed by one source or another, and thus difficult to isolate and measure the contribution of the public’s advice among all the input received.

Unbiased evaluation of any program or project is never easy and public engagement projects are no exception. One reason is that not all sponsors of programs or projects genuinely want to know about failures since these can have adverse consequences. Sometimes, projects fail because of failures to implement a high quality group process, and the advice received becomes unreliable. However, in a democratic society where the people are the final arbiters, it can be even more difficult to reach and accept conclusions that the public’s advice was of little value or that the people’s judgment is wrong.[33] This is especially so if the reason for disagreeing with the public is not because of error or poor quality process but because the public has weighed or valued the decision choices differently than the decision makers. Public will is not the same as political will in these circumstances. Decision makers may be reluctant or unwilling to state this candidly in public or accept such an evaluation result.

At other times, unbiased evaluation is challenging because the participants themselves may be too easily or quickly satisfied with having been given the opportunity to speak without verifying that their opinions carry weight or are seriously considered. Public engagement can make both sponsors and participants feel good even though the tangible benefits may be limited. This “feel good” effect, was frequently observed in these projects. Instances of genuine conversation between people of varied backgrounds and interests are a rare occurrence in the lives of many citizens and leaders today. When it happens, participants often feel uplifted or inspired by the experience. Perhaps simply seeing decision makers present or participating in the meetings is enough to exert a salutary effect independent of the influence exerted on the decision eventually made.

Likewise, leaders and decision makers who participate or observe high quality public engagement sessions appear to come away thinking positively about the value of engaging the public even before the advice has had a real opportunity to make a difference in shaping their pending decision. Because of these phenomena, it can be tempting for sponsors of public engagement projects to declare projects a success simply because advice was sought and obtained and the participants reported being satisfied. However, talk itself does not constitute action, and while the effects on participants and sponsors are not trivial, the ultimate test of a successful public engagement exercise must lie in the consideration given or the use made of the advice obtained.

Lastly, some of the outcomes posited from participating in a public engagement exercise involve changes in the participants such as increases in awareness level or understanding of an issue, an increased sense of agency and well-being, and increases in public-spiritedness.[34][35] Because some of these potential outcomes are intangible, they are more difficult to measure. Also, any increases in the capacity of participants to act as citizens may not be realized until a later time on a completely different issue which cannot be captured by evaluators.

Possible Approaches Stemming From Lesson # 8

A partial solution to the challenge of evaluation is to conduct public engagement projects with an independent evaluation by a party not directly responsible for the work. However, it can be difficult for outside evaluators to disentangle the inner workings of a government bureaucracy and to earn the trust of agency personnel. Perhaps another solution would be to have the decision makers state the decision publicly , and to enunciate the reasons for it. If that includes stating that the decision makers have weighed things differently or have found new reasons to make their choice, then the public may be better able to accept the decision recognizing that the public officials who make decisions are officially empowered to do so, and have an obligation to discharge the duties of their office according to their best judgment. But repeated instances of failure to make use of public advice are a warning sign to participants that real commitment to participatory democratic approaches is lacking. However, some instances of such failure of alignment between the public and political wills may be expected in a representative democracy. Capturing intangible or delayed results may require development of more sensitive measurement indices and specially designed longitudinal studies.

Lesson #9:

Reaping the full benefits of engaging the public will require creating an organizational “home” for public engagement within the organizational structure of the agency or sponsor.

None of the projects in this series were carried out under the aegis of an Office of Public Engagement with an assigned mission, staff, and resources and this shortcoming handicapped the implementation of the projects because each project had to be handled separately as a new event.

Possible Approaches Stemming From Lesson # 9

Creating an Office of Public Engagement within an agency would communicate the importance the agency attaches to the collaborative work and principles of democratic practice to the leadership and staff still unfamiliar with participatory approaches. It would also assist in detecting pending decisions which are difficult or stalemated, help assure an ongoing funding stream, create a cadre of experienced staff to help support projects throughout the agency, and document institutional memory from project to project. Also, when agencies are more willing to embrace principles of shared decision making by creating an organizational “home,” then agency staff can invoke the agency commitment and are likely to find it easier to implement public engagement.

Lesson #10:

Collaboration with organizations outside of government helps to implement projects effectively and expeditiously.

Public engagement projects require different forms of experience and expertise in order to be carried out successfully. Examples of the varied competencies and skill sets needed are:

1) Obtaining genuine agency buy-in for public engagement,
2) Group process design,
3) Generating decision options and criteria,
4) Expertise in the subject area which is the focus of the engagement process,
5) Neutral facilitation skills,
6) Logistics management,
7) Participant recruiting,
8) Communication with top-level decision makers not directly involved in the process, and
9) Independent evaluation.
Some of these functions are best carried out by the staff of the sponsoring agency with training and experience in public engagement, some by outside contractors who are public engagement specialists, and some can be carried out by internal or external staff.

Possible Approaches Stemming From Lesson # 10

Because of the regulations and administrative obstacles which now exist within government agencies, it can be advantageous to give the management lead for the project to one or more organizations outside the government. This can minimize the number of approvals and bureaucratic requirements which must be met to execute effectively. For example, in this series of projects, the bulk of the materials needed for the projects before and after the event were placed on the website of the Keystone Center rather than on CDC sites. Also, project reports were published and posted on the Keystone site without requiring agency wide approval since the reports were the fruits of the collaborating network and not of one agency in particular. Other requirements within the federal system, such as those related to the Federal Advisory Committee Act, are not always interpreted in ways which facilitates rather than impedes public engagement. In these situations, it can be advantageous for the lead management organization to be outside of government and for the federal agency to be a collaborating organization. It is easier to structure such networks if the federal funds supporting the work can be made part of an existing grant or cooperative agreement with an outside party or coordinating center which in turn can arrange with other parties to help provide needed services more efficiently.

Table 1. Lessons Learned and Approaches Suggested.

Conclusion

Exposure to the ideas of deliberative democracy and experience in conducting these public engagement projects over the last decade have revealed several important reasons why they should be pursued by government agencies and other organizations. The public’s demand for more active participation in the decisions which affect them is large and growing, [24][36] and the agencies face many difficult or stalemated policy decisions. These are challenging because they sometime have competing values and because finding common ground and compromise can be difficult. Failure to tackle these decisions means that federal agencies are missing opportunities to address solvable problems. There is compelling evidence that public engagement can produce better decisions, empower participants, and build much needed public trust in government and help to better achieve agency missions. [7][33]

Public engagement also has the power to be build capacity. That is, to increase the participants’ sense of agency and control over their lives [8]. This sense of control promotes health and a sense of well-being for both the individual participants and the social groups to which they belong. [29] As stated by The National Research Council, “When done well, public participation improves the quality and legitimacy of a decision, and builds the capacity of all involved to engage in the policy process…public participation should be fully incorporated into environmental assessment and decision-making processes, and it should be recognized by government agencies and other organizers of the processes as a requisite of effective action, not merely a formal procedural requirement. [7]”

The Guiding Principles and the DPET model with its inclusion of both citizens-at-large as well as stakeholders provide both a framework and a proven, feasible model which agencies or non-governmental organizations can use to be more effective at working through difficult values-based policy decisions to achieve greater public good.

Appendix A: Guiding Principles for Conducting Decision-focused Public Engagement

1. The desire for advice is genuine and the commitment to an authentic participatory approach is real.

Comment: The government agency or sponsor of the public engagement has a genuine desire and a commitment to learn, listen, and obtain advice from the public, and is willing to place one or more decisions on the table for authentic discussion and deliberation.

2. The decision being considered is suitable and made clear.

Comment: The types of decisions most suitable for decision-focused public engagement are public policy decisions which are governed primarily by values choices. They are decisions for which knowledge of facts is essential, but not sufficient. Not suitable for public engagement are decisions of a highly technical nature, such as purely scientific or largely fact-based decisions. Also, the decision should be an important one with broad and important consequences for the population, to incentivize the public to participate. The values-based and consequential decision should be clearly identified and articulated by the sponsor.

3. The decision-making process is suitable and made clear.

Comment: The characteristics of a decision making process most suitable for public engagement include allowing adequate time to authentically engage the public and adequate advance thinking by the sponsor to offer possible decision options to the public for consideration. Not suitable are processes which have to produce decisions very quickly, have already been closed, or are still so early in the process that the contributions sought are of framing, brainstorming, or researching an issue rather than developing options, weighing pros and cons, and selecting the best option.
For suitable processes, the timeframe, key participants, principal decision maker(s), and the anticipated end product which communicates the decision are made clear and transparent for all to understand.

4. Both non-partisan citizens-at-large and partisan stakeholders are included.

Comment: The goal of a decision-focused public engagement is to obtain an impartial, fully-considered decision that best serves the common good.
Citizens-at-large are the repository of the public’s core values and as such are inherently the most qualified “experts” on the values which will be traded-off in making the public policy choice. Furthermore, their inherent “disinterestedness” on specific issues means they have the least conflict of interest and provide the best means of arriving at the impartial judgments needed to make difficult choices most compatible with the common good.
Representatives from stakeholder organizations are essential to obtain a full understanding of the interests and of the consequences of the decision on the most affected segments of the public.

5. A critical mass and diverse group of participants are recruited to take part.

Comment: An adequate number of citizens-at-large who are geographically representative of the area of interest and demographically representative of the population of interest by age, race, and sex are recruited to participate. A smaller but still comprehensive number of representatives from stakeholder organizations in all the key interest sectors are also recruited to participate.

6. Staff and sufficient resources are committed to the process.

Comment: Agency staff members are active in the project to answer questions, hear the input of citizens-at-large, participate in the stakeholder portion of the consultation, and convey the findings or recommendations to the final decision makers. The agency or sponsor provides adequate resources to support the public engagement process at the level required to meet the purpose of the exercise.

7. Essential facts and unbiased information are provided.

Comment: The key facts required for participants to have a well-informed discussion are provided at the outset of the group process. The facts are presented by one or more experts who are effective speakers. The presentations are made in large group sessions in a fair and balanced manner so that participants share a common set of facts.

8. Conversations are structured and facilitated to produce candid dialogue.

Comment: The overall group process in small and large group sessions is convened and managed in a neutral fashion by an independent facilitator. Ground rules for interaction are agreed to which respect all the participants . The conversations are structured so that participants engage in respectful listening, exchange perspectives relevant to the decision(s), and learn from each other. Active participation by all parties is desired and encouraged.

9. Difficult choices are weighed and recommendations made.

Comment: Participants weigh the choices and make difficult tradeoffs between competing values to help make a decision. Participants do not seek consensus per se but work through their differences to reach general agreement. This agreement is expressed as collective recommendations concerning the pending decision. These results from both publics are summarized and presented as the public’s perspective in a report conveyed to the government agency or sponsor.

10. The recommendations receive “serious consideration” and participants obtain candid feedback about the final decision made.

Comment: At a minimum, the work of the public is conveyed accurately, and receives serious consideration by the decision makers. The agency or sponsor is accountable and provides feedback to the participants about the final decision made and the main reasons for it. Ideally, a robust evaluation of the process and the outcome is carried out to continuously improve the process for the future.

Appendix B: Key Features of the Decision-focused Public Engagement Table (DPET) Model

Participants:
Target number of 100 citizens-at-large
Participants selected from the four geographic segments of the population
Participants representative of their area by age, race, and sex
Normally 20-30 stakeholders from approximately 10 key sectors of interest
Includes agency or sponsor’s staff persons as stakeholders

Meetings:

Four day long dialogue and deliberation sessions for citizens
Stakeholders meet twice separately from the citizens-at-large, once before and after
Stakeholder meeting activities similar to those for citizens-at-large
Stakeholders can provide initial input in framing issues and designing process
Some stakeholders participate as observers at the larger citizens’ meetings
One or two citizens invited from each area to attend final stakeholder meeting

Information presented:

Participants in general session hear an oral presentation in easy to understand language
Effective, non-condescending expert/lecturer presents the minimum amount of unbiased information needed to have an informed discussion of the issue at hand
Opportunities are given to ask questions about the factual information presented
Subject matter experts are on hand to answer questions but not to participate directly in the discussions
Booklet summarizing the key facts needed to have the conversation is presented in a user-friendly fashion
A discussion guide summarizing the choices faced is presented to the participants for use during small group table discussions and large group exchanges.

Dialogue and deliberation:

Neutral facilitation is provided
Participants in small groups listen respectfully, exchange experiences and viewpoints, and deepen their understanding of the issue on the table
Participants in general session listen and consider the views expressed by participants of other small groups and seek clarifications where necessary
Participants weigh the alternative courses of action brought forth at the meeting
Participants as a group vote or otherwise make their preferences known.
Opportunities are given for participants to react to the group findings and to introduce modifications if needed and agreed to by the group.

Product and closure:

Stakeholders consider citizen-at-large input, develop their own views, and synthesize the viewpoints into the “societal perspective” on the decision being placed on the table.
Agency or sponsor participants convey results to key decision makers.
Decision makers provide feedback about the final decision to participants

References

1. Bernier RH, Wills-Toker C. Case abstracts of a multi-year, multi-project public engagement initiative to better inform governmental public health policy decisions. J Participat Med. 2014 May 21; 6:e7. Available at: https://participatorymedicine.org/journal/?p=3407. Accessed May 21, 2014. ↩
2. Feudtner C, Marcuse EK. Ethics and immunization policy: promoting dialogue to sustain consensus. Pediatrics. 2001; 107: 1158-1164. ↩
3. Vaccines and issue of trust. Consumer Reports. August 2001: 17-21. ↩
4. Bulling D, DeKraai M. Evaluation of a Multistate Public Engagement Process on Pandemic Influenza. J Participat Med 2014 Mar 8; 6: e5. ↩
5. Sandy Heierbacher, et al. Engagement Streams Framework. National Coalition for Dialogue and Deliberation. Available at www.ncdd.org/streams Accessed January 11, 2014. ↩
6. Engaging citizens in policy making: Information, consultation and public participation. Organization for Economic Co-operation and Development. Puma Policy Brief Number 10; July 2001. ↩
7. Dietz T, Stern PC, eds Public Participation in Environmental Assessment and Decision Making. Washington DC: National Academies Press; 2008. ↩
8. Yankelovich D. A missing concept. Kettering Review. 1991; Fall: 54-66. ↩
9. Barber BR. A failure of democracy, not capitalism. New York Times; July 29, 2002. ↩
10. Innes JE, Booher DE. Reframing public participation: Strategies for the 21st century. Planning Theory and Practice. 2004; 5: 419-436. ↩
11. Going public [editorial]. Nature. 2004; 431: 883. ↩
12. Boyle P. Public problems, values, and choices. Popular Government. 2001; Fall: 18-23. ↩
13. Bennett M, Gibson JM. A Field Guide to Good Decisions. Westport, CT: Praeger; 2006. ↩
14. Yankelovich D. Speaking truth to power. Renewable Resources Journal. 2004; Summer: 20-22. ↩
15. Obama, B et al. Now Is The Time: The President’s Plan to Protect Our Children and Our Communities by Reducing Gun Violence. The White House. Available at: http://www.whitehouse.gov/sites/default/files/docs/wh_now_is_the_time_full.pdf Accessed January 11, 2014. ↩
16. Yankelovich D. Eighteen propositions for citizen engagement. Address to W.K. Kellogg Foundation Devolution Initiative. Ann Arbor, MI. June 1998. ↩
17. Alinsky SD Rules for Radicals Vintage Books New York 1989.Bozeman B, Sarewitz D. Public values and public failure in U.S. science policy. ↩
18. Bozeman B, Sarewitz D. Public values and public failure in U.S. science policy. Science and Public Policy. 2005; 32: 1-18. ↩
19. Helps A. Phrase Finder. Available at: http://www.phrases.org.uk/meanings/261100.html Accessed January 11, 2014. ↩
20. Public engagement pilot project on pandemic influenza. National Coalition for Dialogue and Deliberation. Available at: http://ncdd.org/rc/item/1448. Accessed January 11,2014. ↩
21. Prince Shotoku [quotation]. Available at: http://www.greatthoughtstreasury.com/prince-sh%C5%8Dtoku-born-shotoku-taishi-aka-prince-umayado-or-prince-kamitsumiya/quote/decisions-importan Accessed January 11, 2014. ↩
22. DelliCarpini MX, Cook FL, Jacobs LR. Public deliberation, discursive participation, and citizen engagement: A review of the empirical literature. Annu Rev Polit Sci. 2004; 7:315-344. ↩
23. Rowe G., Fruwer LJ. A typology of public engagement mechanisms. Science, Technology and Human Values. 2005; 30: 251-290. ↩
24. Arnstein SR. A ladder of citizen participation. Journal of the American Planning Association. 1969; 35: 216-224. ↩
25. International Association for Public Participation. Spectrum. Available at: http://www.iap2.org/associations/4748/files/spectrum.pdf Accessed January 11, 2014. ↩
26. Stern PC, Fineberg HC, eds Understanding Risk: Informing Decisions in a Democratic Society. US National Research Council. National Academy Press. Washington DC 1996. ↩
27. Pateman C. Participatory democracy revisited. Perspectives on Politics. 2012: 10: 7-19. ↩
28. Lukensmeyer CJ. Bringing Citizen Voices to the Table: A Guide for Public Managers. Jossey Bass, San Francisco, CA. 2015. ↩
29. Marmot M. Health in an unequal world. Lancet. 2006; 368: 2081-2094. ↩
30. Urbina I. Beyond beltway, health debate turns hostile. New York Times. August 7, 2009. Available at: http://www.nytimes.com/2009/08/08/us/politics/08townhall.html?_r=0 Accessed January 11, 2014. ↩
31. Homeland Security Council. National Strategy for Pandemic Influenza. November 2005. Available at: http://www.flu.gov/planning-preparedness/federal/pandemic-influenza-strategy-2005.pdf Accessed January 11, 2014. ↩
32. Department of Health and Human Services and Department of Homeland Security. Guidance on allocating and targeting pandemic influenza vaccine. 2008. Available at: http://www.flu.gov/images/reports/pi_vaccine_allocation_guidance.pdf Accessed January 11, 2014. ↩
33. Dietz T. Bringing values and deliberation to science communication. PNAS 2013; 110: 14081-87. ↩
34. Mansbridge, J. Does participation make better citizens?” Paper presented at the Political Economy of a Good Society (PEGS) Conference, Evanston, IL, February 11-12, 1995. Available at: http://www.cpn.org/crm/contemporary/participation.html. Accessed January 11, 2014. ↩
35. Barry E. With citizenship comes a passion for politics and life. New York Times. July 28, 2007. Available at: http://www.nytimes.com/2007/07/28/nyregion/28citizen.html?_r=2&n=Top%2FReference%2FTimes%20Topics%2FSubjects%2FI%2FImmigration%20and%20Refugees& Accessed January 11, 2014. ↩
36. Applegate JS. Beyond the usual suspects: The use of citizens advisory boards in environmental decisionmaking. Indiana Law Journal. 1998; 73: 903-957. ↩

Copyright: © 2014 Roger Harvey Bernier. Published here under license by The Journal of Participatory Medicine. Copyright for this article is retained by the author, with first publication rights granted to the Journal of Participatory Medicine. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution 3.0 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.