Editors’ Note: This article is one of two companion papers published simultaneously in this Journal. Both papers examine a CDC initiative on public engagement and public policy decision making.

Abstract

Summary: A public engagement initiative was undertaken in 2001 to find ways in which the Centers for Disease Control and Prevention (CDC) could work productively with the public to better inform decision making in the short term and improve trust over the longer term. A new model, renamed the Decision-focused Public Engagement Table (DPET), was developed and used in nine projects between 2005 and 2009. The origins of this initiative, the characteristics of the DPET model, and detailed case abstracts describing the key features and conclusions from each project are described in this paper.

While there was variation in how the model was replicated, its essential features demonstrated that it is feasible and practical to carry out for a variety of difficult decisions involving different agency sponsors. The degree to which public advice influenced decisions was mixed. For some projects the advice was clearly used to shape the decision making, for others it was difficult to isolate the public’s influence from that of other contributors, and for some projects the public input was not seriously considered.

The project evaluations showed an increase in trust, at least in local government, at the time the projects ended, and citizens reported they were more likely to participate in other types of engagement in the future. Taken as a whole, the projects provided “proof of principle” that engaging the public in participatory policy making sponsored by governmental agencies is feasible and can be influential. The initiative also provided clear evidence that citizens make policy choices which are different from those made by experts on the same issue. And, they appear to be better decisions because they are both well-informed and well-aligned with public values and the broader public interest.

Despite the success achieved, the projects did not result in the creation of any permanent infrastructure to support and strengthen public engagement work within the participating agencies. This was disappointing and the reasons are numerous and complex. They are the subject of a companion paper being published in this issue of the Journal which examined the challenges encountered from a broader perspective.[1] It also includes possible approaches to overcome the challenges and embed public engagement as an effective means of addressing the frequent gridlock caused by difficult, values-based public policy decisions.
Keywords: Public engagement, decision-focused public engagement table (DPET), pandemic influenza planning.
Citation: Bernier RH, Wills-Toker C. Case abstracts of a multi-year, multi-project public engagement initiative to better inform governmental public health policy decisions. J Participat Med. 2014 May 22; 6:e7.
Published: May 22, 2014.
Disclosures: The authors have declared that no competing interests exist. The analysis and opinions expressed here are solely those of the authors and not the official views of the CDC or any other government or non-governmental organization. The authors were employed at the CDC or by one of its contractors during the period of time the projects were carried out. The information used for this review was obtained from existing project reports available publicly online, and from published and unpublished manuscripts. Acknowledgments and citations have been provided where appropriate in this manuscript and in a companion paper published in this issue of the Journal [1]

Introduction

Prompted by the controversy over a possible relationship between vaccines and autism and the lack of trust it reflected between the federal Centers for Disease Control and Prevention and some segments of the public,[2] an initiative was undertaken in 2001 to find ways in which the CDC could work productively with the public and, in so doing, build greater trust. After reviewing the types of work conducted by the agency, it was determined that active public engagement in policy making about vaccines was the most suitable work for sharing with citizens. This choice was made because science policy decisions often involve difficult choices between competing values. As the repository of our core public values, citizens are the ones who hold and best understand the values with which our policies should be aligned in making public decisions. As stated by Yankelovich, public engagement can help identify the “boundaries of political permission. [3]”

An initial meeting was held at Wingspread in Racine Wisconsin in July, 2002. It consisted of an ad hoc group of approximately 40 stakeholders representing the full breadth of key interests around vaccines. A follow-up meeting was held in Denver in 2003. The Wingspread group developed a new model for engaging the public which included both citizens-at-large as well as representatives of stakeholder organizations, including the sponsoring governmental agency.[4][5] The new model was piloted on a difficult, values-based national policy decision being faced by the CDC and its parent organization, the Department of Health and Human Services. This first project was an inquiry about what the federal government should recommend to state and local health departments nationwide on the issue of who should be vaccinated first in a severe pandemic of influenza when supplies of vaccine are expected to be limited. This was a difficult rationing type decision requiring a basic understanding of pandemic influenza and the weighing of many equivalent or nearly equivalent values. Knowledge about influenza was essential but not sufficient for making the decision, and thus citizens were felt to be qualified to participate.

Conducted in 2005 as the “Public Engagement Pilot Project on Pandemic Influenza” or (PEPPPI), the project succeeded in providing “proof of principle” for the vaccine and public health communities that engaging the public on a values-oriented policy decision by a federal agency setting policy for the nation is both possible and productive (see case abstract #1).

Because of the success of the first project, an additional project was soon commissioned in 2006 by the Coordinating Center for Infectious Diseases at CDC on the question of what community control measures for pandemic influenza (eg, closing schools early in a pandemic, asking sick people and their well contacts to stay at home, and cancelling football games and other mass events) would be acceptable. It was explained that the control measures would cause social disruption in order to slow the spread of the disease and thus avoid overwhelming hospitals and other services, and these benefits were to be weighed against the disruptive social costs. This project was selected as the co-winner of the Project of the Year award from the International Association for Public Participation in 2007 (see case abstract #4).

Based on the early successes of these two projects, a series of 7 other public engagement projects has been undertaken by the CDC or the National Vaccine Program Office (NVPO) in the three year period 2007-2009. These have each sought to make use of or modify the new joint model of citizen and stakeholder public engagement now called a Decision-focused Public Engagement Table (DPET). This model can be characterized as having or seeking to implement the following key features:

1. A focus on undecided policy choices
2. Inclusion of representative groups of both stakeholders and citizens-at-large
3. Linkage with government decision makers
4. Neutral facilitation
5. Presentation of balanced information from credible sources
6. Frank dialogue and genuine deliberation
7. Reaching some form of group judgment or agreement
8. Synthesis of the results as the “public perspective” on the pending decision
9. Feedback to the participants

Over the decade since the completion of the work by the Wingspread group, a total of nine projects have been carried out including two projects which each tackled the question of who first to vaccinate in a pandemic. The project topics and titles were:

• The First Public Engagement Pilot Project on Pandemic Influenza Vaccine Priorities
• The Public Engagement Project on Community Control Measures for Pandemic Influenza
• A Federal Agency Public Engagement Project on Objectives and Prioritization Criteria
• The Second Public Engagement Project on Influenza Vaccine Priorities
• Use of Public and Stakeholder Engagements to Inform
• National Guidance on At-Risk Populations and Pandemic Influenza
• The Public Engagement Project on Setting Priorities for CDC Vaccine Safety Research
• The Public Engagement Six-State Demonstration Project
• Public Engagement on the National Vaccine Plan
• Public Engagement Project on the H1N1 Pandemic Influenza Plan

Most of these projects were evaluated as they took place and afterwards by the independent, non-profit Public Policy Center at the University of Nebraska. Separate evaluation reports were issued for several projects evaluated and a separate paper summarizing them has been published.[6]

Case Abstracts of Public Engagement Projects

Case Abstract #1: Citizen Voices on Vaccine Choices: The First Public Engagement Pilot Project on Pandemic Influenza

 
Performance period: July 2005- December 2005.

Background and question addressed:

The decision over how best to allocate the relatively small supplies of vaccine likely to be available in the early months of a severe pandemic is a difficult one because it has to exclude some persons who could benefit from vaccination and might die without it. This public policy decision is not only a question of what the science dictates, but it also involves a consideration of public values.

In order to better inform federal guidance on vaccine prioritization and to evaluate a new model for engaging the public on values-laden policy decisions, the Centers for Disease Control and Prevention (CDC) recruited citizens-at-large with no declared agendas and stakeholders with recognized points of view for day-long dialogue and deliberation sessions.

The key question was: who should be vaccinated first in the first months of a pandemic when supplies are limited?

Structure and process.

This project was the first carried out by:

• Obtaining a representative national sample of both lay citizens and stakeholders from various interest groups;
• Giving them basic information needed to have an informed discussion;
• Engaging them in give and take discussions with fellow participants;
• Providing the opportunity to weigh tradeoffs between different choices; and
• Asking them to determine the preferred course of action by voting for their top three goals for a vaccination program given five goals to choose from.

These objectives were pursued using the following methods: (1) two day-and-a-half meetings with 50 national stakeholders, (2) one day-long consultation with 101 citizens in the south (Atlanta), and (3) three half-day feedback sessions with 165 citizens in the east (Massachusetts), Midwest (Nebraska), and west (Oregon).

Participating organizations.

Fourteen organizations collaborated on the project, including:

• Atlanta Journal Constitution

Georgia Department of Human Resources—Division of Public Health

• Institute of Medicine
• Massachusetts Health and Human Services
• National Immunization Program, CDC
• National Vaccine Program Office, DHHS
• Nebraska Health and Human Services
• Oregon Department of Human Services
• Practicum Limited
• Richard Lounsbery Foundation
• Study Circles Resource Center
• The Keystone Center
• University of Georgia
• University of Nebraska Public Policy Center

Facilitation: The Keystone Center.

Type of contributions by participants:

Learning about influenza, exchanging different viewpoints with other participants, weighing different values or goals, and reaching judgment about best course of action. In effect, the participants acted as a “policy jury” rendering their verdict on the best course of action.

Outcome of the project:

All publics decided with a very high level of agreement that assuring the functioning of society should be the highest priority immunization goal, assuming a moderate to severe 1918-like influenza pandemic. This goal was followed in importance by the goal to reduce individual deaths and hospitalizations. There was little support for the other proposed goals to vaccinate young people first, use a lottery system, or take a first-come-first-served approach.

Use/impact:

The initial guidance issued by the Department of Health and Human Services in November 2005 was based primarily on the advice of experts in medicine and public health. It prioritized reducing individual deaths and hospitalizations and was different from the advice of the public from this first project. However, this discrepancy and the pushback received persuaded the Department to re-engage the public. It issued final guidance in 2008 reflecting many of the public’s recommendations from the second project (see below) and reaffirmed the public’s recommendations from the first project, including the advice to place higher importance on protecting those individuals who assure the functioning of society (critical occupations) rather than those at highest risk because of illness or disease.

Conclusion for public engagement:

As the first test of the new public engagement model, this project provided “proof of principle” for the vaccine and public health communities that engaging the public on a values-oriented policy decision by a federal agency setting policy for the nation is both possible and productive. It also demonstrated in conjunction with a second vaccine prioritization project that the general public weighs policy choices differently than the experts.

Formal evaluation: Yes, by the University of Nebraska Public Policy Center.

Link to project report:

http://ppc.unl.edu/wp-content/uploads/2005/12/PEPPPI_FINALREPORT_DEC_2005.pdf
Accessed April 2014.

Link to evaluation report:

See Attachment L in the final report for the methods and results of the evaluation. Also, a summary of the evaluation is presented in the main body of the report.

http://ppc.unl.edu/wp-content/uploads/2005/12/PEPPPI_FINALREPORT_DEC_2005.pdf
Accessed April 2014.

Acknowledgements.

More than 200 citizens-at-large in Georgia, Massachusetts, Nebraska, and Oregon attended meetings and engaged in productive and respectful dialogue and deliberation and worked together to provide their best advice.

A group of 50 stakeholder-citizens did likewise. Their names and organizational affiliations are listed in Attachment A in the final report (see link above).

A writing group made up of some of the leaders and organizers of the project prepared a draft manuscript describing the project results. They included: Jon Abercrombie, Roger Bernier, Victoria Evans, Mary Davis Hamlin, Tanya Johnson, Matt Leighninger, Edgar Marcuse, and Caitlin Wills-Toker.

Case Abstract #2: The Public Engagement Project on Vaccine Prioritization for Pandemic Influenza

 
Performance period.
2006 to 2008.

Background and question addressed:

In 2006, an interagency workgroup sponsored by the Department of Health and Human Services and the Department of Homeland Security was established to revisit initial recommendations made in November 2005 and set forth new national guidance. The workgroup sponsored public consultations of both nonpartisan citizens-at-large and stakeholders from organizations with recognized interests as one stream of input to identify the highest priority goals for a vaccination program when supplies are expected to be limited. The assumption made was that the anticipated pandemic would be of the severe variety, akin to the 1918 influenza pandemic.

Structure and process:

The project was conducted in two phases of day-long dialogue and deliberation sessions—one before the guidance was drafted and another as part of the vetting process to obtain suggestions for revisions. In the first phase, there were two citizens-at-large meetings in Nassau County New York and Las Cruces New Mexico and one stakeholder meeting in Washington DC with 100+ participants at each meeting. Attendees were asked to rate ten proposed goals on a 7 point Likert scale. The second phase meetings to solicit public response to the guidance were held in Milwaukee, Wisconsin, Hendersonville, North Carolina, and Washington DC. Altogether almost 500 citizens and 80 stakeholders participated.

In addition to face-to-face meetings, the project sponsored a web-based dialogue to obtain public input on the draft guidance from a larger segment of the public than might be engaged through in-person meetings.

Lead sponsor: Department of Health and Human Services.

Participating organizations:

Altogether, the project involved the collaboration of 17 organizations. They include:

• Association of State and Territorial Health Officials
• Border Health Education Training Center at NM State University
• Centers for Disease Control and Prevention
• City of Milwaukee Health Department
• Henderson County Department of Public Health
• Nassau County Department of Public Health Emergency Preparedness
• National Association of County and City Health Officials
• New York State Senator Michael Balboni
• New York State Department of Health
• New Mexico State Health Department
• North Carolina Department of Health and Human Services
• Southern Area Health Education Center Environmental Health Comm
• The Keystone Center
• University of Nebraska
• US Department of Homeland Security
• Wisconsin Department of Public Health

Facilitation: The Keystone Center.

Type of contributions by participants:

Learning about influenza, exchanging different viewpoints with other participants, weighing different values or goals, and rating each of ten proposed goals for a vaccination program.

Outcome of the project:

In the first phase, the goals to vaccinate persons who are (1) essential to implementing a pandemic response program, (2) provide essential services in communities, (3) are at increased risk of pandemic influenza because of their jobs, and (4) children received the highest mean scores of 6.5, 6.0, 5.8, and 5.5 respectively. Vaccination of persons that are at highest risk of severe pandemic illness or death was ranked seventh with a mean score of 4.7. In the second vetting phase of the meetings, 69-93% of the citizens and stakeholders agreed or strongly agreed with the goals selected in the first phase, and 57-83% agreed or strongly agreed with the overall prioritization guidance. Several substantive suggestions for revisions to the guidance, such as including all persons in the vaccination program regardless of citizenship status were made.

Use/impact:

The results from this project were used by the interagency workgroup to help create and finalize an overall strategy for simultaneous (as opposed to sequential) vaccination of the highest rated groups within four pre-selected categories of important groups and for making revisions to the draft guidance. The highest rated priority group (critical occupations) identified in this project were similar to those identified in an earlier public engagement project, but different from that recommended in the initial HHS guidance (high risk persons) based on input from experts.

Conclusion for public engagement:

In demonstrably helping to shape national policy on vaccine prioritization, this project has provided further proof of the effectiveness of public engagement as a means of making better public policy decisions. This effectiveness was highlighted by HHS Secretary Mike Leavitt in a July 2008 press release announcing publication of the “Guidance on Allocating and Targeting Pandemic Influenza Vaccine.” He stated, “This guidance is the result of a deliberative democratic process…All interested parties took part in the dialogue; we are confident that this document represents the best of shared responsibility and decision making.”

Formal evaluation: Yes, by the University of Nebraska Public Policy Center.

Link to project report:

Draft Final Report from the in-person meetings prepared by the Keystone Center entitled “Public Engagement Project on Vaccine Prioritization for Pandemic Influenza—Final Report” is available upon request to the corresponding author (RH Bernier, rbernier@uga.edu).

Final Report from the web dialogue prepared by WestEd, Jones and Stokes, and Ross & Associates Environmental Consulting, Ltd entitled “Prioritization of Pandemic Influenza Vaccination—Evaluation of an Online Dialogue is available upon request to the corresponding author.

US Department of Health and Human Services Guidance Document:
http://www.flu.gov/images/reports/pi_vaccine_allocation_guidance.pdf
Accessed April 2014.

Link to evaluation report.
http://ppc.unl.edu/wp-content/uploads/2008/02/Pandemic-Infuenza-Evalaution-Report.pdf
Accessed April 2014.

Acknowledgements:

More than 200 citizens-at-large in New York, New Mexico, Wisconsin, and North Carolina and a group of 90 stakeholders in Washington DC attended meetings and engaged in productive and respectful dialogue and deliberation and worked together to provide their best advice.

A writing group made up of some of the leaders and organizers associated with one or more of the above organizations prepared a draft manuscript describing the project results. They include: Anna Buchanan, Bruce Gellin, Don Greenstein, Til Jolly, Benjamin Schwartz, Tom Shimabukuro, Matthew Somes, Toscha Stanley, and Doug Thompson.

Case Abstract #3: A Federal Agency Public Engagement Project on Objectives and Prioritization Criteria (The Goals Project)

 
Performance period: August to December 2006.

Background and question addressed:

In the fall of 2006, the Centers for Disease Control and Prevention consulted with the public to better understand and take into consideration the public’s values and perspectives in formulating strategic objectives and priorities in four health goal areas. The project had two primary purposes, namely 1) to determine the extent of support, if any, for a proposed set of four criteria to be used in choosing the highest priority objectives from a “starter list” of 80 possible objectives, and 2) to identify which individual objectives were the highest priority objectives for both citizens and stakeholders. A secondary purpose was to learn more about a new model for engaging the public to improve the quality of the agency’s future interactions with the public.

Structure and process:

The project was carried out principally in a series of meetings with citizens-at-large and stakeholders. Small group roundtable discussions and large group sessions were held to discuss the proposed criteria and potential objectives. Organizations in partnership with the CDC in San Antonio, Texas, Oakland, California, Boston, Massachusetts, and Little Rock, Arkansas were used to recruit persons known to the associated organizations. An average of 38 persons attended the four citizens-at-large meetings. A total of 106 organizations sent stakeholder representatives to a meeting in Washington DC. A Task Force convened by Partnership for Prevention and made up of members from partner organizations was created to synthesize the input received and make recommendations to the sponsor.

Lead sponsor: The Centers for Disease Control and Prevention.

Facilitation: The Keystone Center.

Type of contributions by participants

Learning about the CDC goals and objectives, exchanging different viewpoints with other participants, and selecting priority objectives through an electronic polling technique.

Outcome of the project:

Overall an estimated 300 participants were engaged at the public participation events with a majority of the citizen attendees working in public health or the health care system or in a closely allied field. Participants agreed with the four priority setting criteria proposed by the agency for establishing the highest priority objectives. Furthermore, the participants identified 24 specific objectives from the starter list as their highest priorities related to all four of the agency’s main goals. In addition, the Task Force made 11 recommendations to the agency based on overarching themes uncovered during the public engagement sessions.

Use/impact:

The impact of this public consultation on agency decision-making appears to have been nil on the main question about prioritization and very limited overall. Over two years after the report of the Task Force, the CDC website still listed all of the initial 80 starter objectives, indicating that no list of high priority objectives was ever prepared or finalized. The agency abandoned or never implemented its original intent to prioritize objectives.

Conclusion for public engagement:

The strengths of the project included the number of sources used and the quality of the neutral facilitation. However, the limitations on the findings were numerous and serious, including inadequate representation of citizens-at-large. The process of informing and presenting information about the objectives to the participants varied and did not really allow adequate comparison between objectives. The amount of time allotted to deliberate was inadequate given the large number of objectives that needed to be assessed. Participants were allowed to vote on objectives they had not learned about in small group discussions which means that some of the opinions expressed were not as well considered as others. The Task Force concluded in its report “the results from these engagements cannot be generalized to any larger group.”

As an early test of a new public engagement model, the failures of this public consultation have more to teach us than its successes. In particular, agencies embarking on public consultations need to assure commitment to the project and its implementation prior to initiating public engagement projects, and make a commitment to give serious consideration to the input received. The causes of failure to implement this projectwere preventable and should have been addressed in advance of the project.

Formal evaluation:

No independent research evaluation of the process or the outcome was carried out. However the Task Force report included several observations about the quality of the project and recommendations for future directions.

Link to project report:

The Report of the Partners’ Task Force on Objectives, Partnership for Prevention, 1015 18th St NW, Washington DC (2006) is available upon request to the corresponding author (RH Bernier, rbernier@uga.edu).

Link to evaluation report: See project report from the Task Force.

Acknowledgements:

See lengthy list of acknowledgements in the Task Force report for CDC staff members, senior executives from CDC, state and local health officials, leaders in academic medical centers, the staff of the Keystone Center, and the staff at Partnership for Prevention.

A writing group made up of some of the leaders and organizers associated with one or more of the above organizations prepared a draft manuscript describing the project results. They include: Mike Hughes, Peter Rzeszotarski, and Roger Bernier.

Case Abstract #4: The Public Engagement Project on Community Control Measures for Pandemic Influenza

 
Performance period: 2006.

Background and question addressed:

The Centers for Disease Control and Prevention set out to engage the public in the fall of 2006 to determine if the public would support implementation of a package of influenza control measures designed to reduce the spread of pandemic influenza while awaiting development and distribution of an effective vaccine. Included in this package of control measures were potentially socially disruptive interventions such as voluntary home quarantine of sick people and members of their households, closing schools, canceling large gatherings and altering workplace schedules. A secondary purpose was to determine what ideas the public had to mitigate the impact of the control measures. The Community Control Measures Project was also intended to expand CDC’s knowledge about how to conduct effective public engagement.

Structure and process:

The project took place in two phases. First , a target group of 100 citizens was recruited from Atlanta Georgia (South), Syracuse New York (East), Lincoln Nebraska (Midwest), and Seattle Washington (West). Second, a target group of 50 national stakeholders held a two-day meeting in Washington, DC.

Lead sponsors and participating organizations:

Association of State and Territorial Health Officials and The Keystone Center were co-sponsors and 14 other organizations participated in the project, including:

• Center for Biopreparedness Education—Omaha
• Centers for Disease Control and Prevention
• FOCUS Greater Syracuse
• Georgia Department of Human Resources—Division of Public Health
• Infectious Disease Society of America
• National Association of County and City Health Officials
• Nebraska Health and Human Services System
• New Jersey Department of Health and Senior Services
• New York State Department of Health
• Public Health—Seattle and King County
• Searcy, Weems-Scott, and Cleare
• The Keystone Center
• United Parcel Center
• US Department of Education
• US Department of Health and Human Services

Facilitation: The Keystone Center.

Type of contributions by participants:

Learning about influenza community control measures, exchanging different viewpoints with other participants, weighing whether or not to support potentially disruptive control measures, deciding on a preferred course of action, and suggesting actions to mitigate the effects of the control measures.

Outcome of the project:

A total of 259 citizens diverse in age, race, and gender from four parts of the US supported implementation of the control measures. There was a high level of agreement for the three measures to keep sick persons at home, change work patterns and schedules, and cancel large gatherings. Although support was still high, participants found asking the non-ill contacts of sick persons to stay at home and closing schools harder to accept. Thirteen recommendations for mitigating the negative social effects of the control measures were forwarded to the decision makers.

Use/impact:

The clear and high level of public support for the federal agency’s proposed recommendations, along with a Harvard public opinion poll carried out around the same time, helped convince the agency and its critics that the control measures would be supported and therefore appearing feasible to implement. The findings from the public engagement were cited in the pre-pandemic planning guidance issued by CDC, however it cautioned that it was uncertain how the public might react when an actual pandemic occurs (The public frequently cited the need for job security and income protection to ensure compliance with control measures). The 13 recommendations for mitigation were not adopted by the agency because it said the development of specific mitigation strategies would be left to local communities.

Conclusion for public engagement:

This project “Putting the Public in Public Health” was named co-winner of the year by the International Association for Public Participation in 2007.
The project confirmed that public engagement on a difficult values-oriented public policy decision is possible and that rendering a verdict of public support can be useful to decision makers, especially when it reveals that the public will is consonant with the political will. This feature of the results may be perceived of greater value than the advice about specific questions posed to the public.

Formal evaluation: Yes, by the University of Nebraska Public Policy Center.

Link to project report:

http://ppc.unl.edu/wp-content/uploads/2007/05/FinalReportMay2007.pdf
Accessed April 2014.

Link to evaluation report:

The full evaluation report is contained in the project report as Appendix B.
http://ppc.unl.edu/wp-content/uploads/2007/05/FinalReportMay2007.pdfAccessed April 2014.

Acknowledgements:

More than 200 citizens-at-large in Georgia, Nebraska, Washington state, and New York attended meetings and engaged in productive and respectful dialogue and deliberation and worked together to provide their best advice.

A group of 35 stakeholder-citizens did likewise. Their names and organizational affiliations are listed in Attachment A in the final report (see link above).

A writing group made up of some of the leaders and organizers of the project prepared a draft manuscript describing the project results. They include: Lisa Koonin, Jacquelyn Polder, and Roger Bernier.

Case Abstract #5: Public Engagement Project on At-Risk Populations and Pandemic Influenza

 

Performance period: 2008.

Background and question addressed:

Not all members of the population will be equally prepared for and capable of withstanding the medical and social consequences of an influenza pandemic. To reduce harms across different segments of the population, it is important to answer the question of who is at increased risk of adverse outcomes and to plan for providing the services the at-risk populations will need to stay as healthy and safe as possible. The Association of State and Territorial Health Officials (ASTHO) worked with public health agencies, federal entities, members of the public, and key stakeholders to develop national guidance on at-risk populations and pandemic influenza, including guidance on who exactly constitutes the special at-risk population during a pandemic.

Structure and process:

Project planners conducted public deliberations with a target sample of 100 community residents representative of the at-risk population by category of risk, age, race, and sex from two areas of the US and with representatives of stakeholder organizations from major sectors likely to be most knowledgeable about at-risk populations. Altogether 144 stakeholders from different interest groups and at-risk community residents participated in the discussions and deliberations.

Lead organizations and sponsors:

Association of State and Territorial Health Officials and the Centers for Disease Control and Prevention and 11 participating organizations, which include:

• Brookline Massachusetts Health Department
• Boston University School of Public Health
• Center for Infectious Disease Research and Policy
• City of Boston Public Health Commission
• Johnson County Health Department
• Kansas City Health Department
• Massachusetts Department of Public Health
• National Association of County and City Health Officials
• One KC Voice/Consensus
• State of Missouri Department of Health and Senior Services
• State of Kansas Department of Health and Environment

Facilitation: The Keystone Center.

Type of contributions by participants:

The group processes were structured to provide essential information to the participants, permit them to engage in give-and-take discussion with diverse participants in small groups, to share key points from their small-group discussions, and to vote for their preferences on key ideas presented for consideration.

Outcome of the project:

More than two-thirds of participants agreed with the concepts included in the proposed definition of at-risk populations and made several proposed wording changes to the definition. Several broadly supported recommendations were made on unmet needs related to information, community preparedness, healthcare, and finances.

Use/impact:

The final national planning guidance issued in August 2008 to state, territorial, tribal, and local health departments incorporated some of the findings from the public consultations directly into the wording or recommendations of the guidance. Other findings were used to identify issues of special concern and these were highlighted in the guidance.

Conclusion for public engagement:

This project provides additional evidence that public engagement methods can be productive when agencies have clarity of purpose, about the decisions they want to make, and are committed to using the results. It provided advice that strengthened the support for the difficult public policy decisions such as who should be included in the definition of an at-risk population.

Formal evaluation: No.

Link to project report:

Project Report—Accessed April 2014:
www.astho.org/Programs/Infectious-Disease/At-Risk-Populations/ARPP-Engagement-Meetings–Final-Report,-Sept-2008/

National Guidance—Accessed April 2014:
www.astho.org/Programs/Infectious-Disease/At-Risk-Populations/At-Risk-Pop-and-Pandemic-Influenza-Planning-Guidance

Link to evaluation report: Not applicable.

Acknowledgements:

More than 100 citizens-at-large in Massachusetts and Kansas City and more than 20 stakeholders attended meetings and engaged in productive and respectful dialogue and deliberation and worked together to provide their best advice.

A writing group made up of some of the leaders and organizers of the project prepared a draft manuscript describing the project results. They include: Anna Buchanan, Caroline Barnhill, Amy Becker LaFrance, and Natalie Vestin.

Case Abstract #6: Public Engagement Project on Vaccine Safety Research

 

Performance period: 2008 to 2009.

Background and question addressed:

A 2005 report from the Institute of Medicine on vaccine safety research and public trust highlighted the importance of transparency in research and recommended that the National Vaccine Advisory Committee (NVAC) review a draft vaccine safety Scientific Agenda developed by the Centers for Disease Control and Prevention (CDC). As part of this review, the NVAC undertook broad public engagement activities to help inform its recommendations to CDC on priorities for vaccine safety research. Given the opposition or criticism to vaccines in some segments of the population, these consultations with an “angry public” were expected to be especially difficult and success was not assured.

Structure and process:

The involvement of the public took place in community meetings in Birmingham Alabama (70), Ashland Oregon (47), and Indianapolis Indiana (52) and at small and large stakeholder meetings in Salt Lake City (27) and in Washington DC (80) respectively.

Lead sponsors:

National Vaccine Program Office and the Centers for Disease Control and Prevention.

Facilitation: The Keystone Center.

Type of contributions by participants

Learning about vaccine safety concerns, exchanging different viewpoints with other participants, weighing different potential criteria for prioritizing vaccine safety research, and selecting criteria to propose to the NVAC.

Outcome of the project:

Two products: (1) Criteria to be used by NVAC in identifying priorities for vaccine safety research described in a draft research agenda from CDC and (2) recommendations for new research which was not included originally on the draft research agenda prepared by CDC.

Use/impact:

The recommendations for criteria from the public engagement sessions were subsequently adopted by the NVAC and used to rank the specific vaccine safety research topics included in the Scientific Agenda. In addition, 32 recommendations were transmitted to CDC for consideration. Furthermore, as reported at a subsequent public meeting of the NVAC, CDC adopted many of the NVAC recommendations regarding vaccine safety research.

Conclusion for public engagement:

The results provide evidence of the capacity of community members and stakeholders with divergent and even polarized views about vaccines to find common ground and work together with proven useful methods productively to co-create guidance useful for federal agencies and their advisory committees.

Formal evaluation: No.

Link to project report:

Click to access nvacrecommendationsisoscientificagenda.pdf

Accessed April 2014.

Link to evaluation report: Not applicable.

Acknowledgements:

Many good faith efforts and courage were put forth by both members of the public as well as health professionals and government officials which made holding the meetings possible in the first place and eventually reaching agreement on a controversial topic.

A writing group made up of some of the leaders and organizers of the project prepared a draft manuscript describing the project results. They included: Kirsten Vannice, Elizabeth Skillen, Janesse Brewer, Daniel Solomon, Anna Buchanan, James Mason, Karen Broder, John Iskander, Roger Bernier, Vicky Debold, Tawny Buck, and Benjamin Schwartz.

Case Abstract #7: Six-State Public Engagement Demonstration Projects on Pandemic Influenza

 

Performance period: 2008 to 2009.

Background and question addressed:

After demonstrating the effectiveness of a new public engagement model in several projects at the national level, CDC awarded grants to six states in 2008 to help local public health officials engage the public in pandemic influenza planning on difficult policy issues of importance in these states. The specific decision to be addressed in consultation with the public was left to the local officials. Awardees were given the option to implement an existing public engagement model such as the one being used at CDC or develop a new approach. Through a competitive process, grant awards were made to Hawaii, Massachusetts, Minnesota, Nebraska, Ohio, and Washington state.

Structure and process:

Some awardees chose to test the CDC model at the local level and others conducted their own approaches to public engagement. The question addressed in each locality and the approach used are described below:

Ohio. Local officials sought to engage the general public and stakeholders in discussions about pending policy decisions related to social distancing strategies to slow the spread of pandemic influenza (Franklin) and how to conduct or modify funerals and burials if large numbers of deaths occurred (Cuyahoga). Two community-at-large public meetings were held in Franklin and Cuyahoga counties followed by stakeholder meetings in each locale one month later.

Washington. Seattle and King county officials sought to better understand community viewpoints on the rationing of health services and how they should be allocated during a severe pandemic when demand will exceed supply (often called “altered standards of care” issue). A diverse group of participants from the general public and key stakeholders used multiple methods to deliberate and provide input, including Q-sort, vignette-driven small group discussions, and pre and post surveys.

Massachusetts. Local officials sought to develop a public information strategy to be implemented in carrying out altered standards of care before and during an influenza pandemic. Three public engagement meetings were held in two areas of the state and two follow up meetings in the same locales a month later using designs developed by local officials and consultants.

Minnesota. The question addressed was how to ethically ration health-related resources, especially vaccine, in a severe influenza pandemic. Two large six-hour community forums including approximately 200 persons were conducted in two separate areas of Minnesota, and nine smaller day-long sessions including 11-16 persons were conducted in the twin cities area and around the state.

Hawaii. The question addressed in this project was –who should receive vaccine in the early days of an influenza pandemic when supplies are limited? Several methods were used to engage the public including community forums, a live, call-in public television series, a newspaper-linked blog, and radio public service announcements.

Nebraska. The project sought to engage six tribes or tribal organizations in identifying a list of the highest priority pandemic influenza planning issues in each of the areas or populations.

Lead organizations and sponsors:

The Centers for Disease Control, the Association of State Health Officials, and the National Association of City and County Health Officials (NACCHO).

Facilitation and logistics: Susan Podziba and Associates, the International City/County Management Association (ICMA), Meridian, and The Keystone Center.

Type of contributions by participants

These varied in each locality depending on the method used. In most of the areas, the degree of citizen involvement was highly interactive involving learning about the issues, exchanging views, deliberating about the difficult choices, and reaching agreement or voting on preferred options.

Outcomes of the project:

Ohio. Thirty-six recommendations from the public regarding non-pharmaceutical interventions for reducing infection rates during an influenza pandemic—eighteen related to school and day-care closures and eighteen related to faith-based communities, including modifications for worship services and end of life special events.

Washington. Citizens endorsed an altered decision making process for the prioritization of scarce, life-threatening medical resources during times of disaster. Participants agreed that prioritization should aim to save the greatest number of people, factoring in survivability of those treated, even if standards of care must be lowered. There was wide agreement that first responders and health care workers should be prioritized, with children and pregnant women given some priority if and when all other factors are equal.

Massachusetts. From the participant input, 26 recommendations were made in four areas, namely trustworthiness of information, increasing compliance with public health advice and guidance, distributing information, and roles for local public health and the community.

Minnesota. Key outcomes were that citizens supported the balance between competing values reflected in the preliminary recommendations of a Minnesota Pandemic Ethics panel but called for changes in how the concept of fairness was presented. Most participants supported age-based rationing at times, however, no single-age ranking scheme was supported by a majority of participants.

Nebraska. The local organizers worked to adapt the key elements of the CDC model to the situation with the tribes. However, no single issue to get public input on was ever selected.

Use/impact:

Ohio. Unclear if public health officials used the recommendations in creating their guidance.

Washington. The findings from this project were presented at the national level to the Institute of Medicine working on developing national guidance, and at the state level in Washington to various groups charged with making recommendations. Public health officials in Seattle King County referenced the findings of this project when making local decisions about H1N1 medical resources. In particular, these public preferences were aligned with local plans for distribution of limited supplies of H1N1 vaccine, and thus knowing the public will was appreciated by the decision makers.

Massachusetts. The public’s recommendations needed to be translated by public health officials into policy decisions that can protect health.

Minnesota. The panel developing ethical guidance reaffirmed its earlier recommendations which the public supported, restated confusing recommendations, and revised recommendations on which there remains uncertainty and disagreement. Also, while not anticipated when the project began, the public’s input was useful in shaping aspects of its H1N1 response which took place later in 2009, particularly guidance about removing barriers to fair access.

Nebraska. There was no collective judgment agreed to about a pandemic planning policy choice and thus no outcome produced that was useful according to the usual measurement of outcomes.

Conclusion for public engagement:

This project demonstrated “proof of concept” that the DPET model or modified versions of it could be successfully implemented below the federal level at least in some locations. Washington and Minnesota, Massachusetts, and Ohio were the most successful in identifying a difficult pending policy issue or issues, and in getting meaningful input. Nebraska and Hawaii created novel approaches that appeared to achieve more limited results. In Nebraska, the tribal cultural values necessitated modification of the DPET model. Where the projects succeeded, the outcomes were often clear, usually in the form of recommendations and these were demonstrably influential at least in Washington and Minnesota. Further use of decision-focused public engagement will require further training of public officials to convey greater understanding about what such work entails, why it is important, and what pitfalls to avoid.

New Resources Created:

In addition to the benefits which accrued to the grantees in each local area, several activities were carried out to inform other state and local areas about public engagement.

A lessons learned workshop was conducted prior to the initiation of the six-state project based on earlier CDC public engagement projects. This report entitled “Lessons Learned from Five Previous Public Engagement Projects” is available as Appendix A page 168 in the University of Nebraska Evaluation Report at the following link: http://ppc.unl.edu/wp-content/uploads/2010/05/P5-Report-FINAL.pdf.

A second lessons learned workshop was convened by ASTHO and NACCHO to identify the lessons learned from the six projects at the implementation level. The report, entitled “Public Engagement Lessons Learned Workshop Summary Report December 2009” is available in the University of Nebraska Evaluation Report as Attachment B at the following link: http://ppc.unl.edu/wp-content/uploads/2010/05/P5-Report-FINAL.pdf.

A planning guide for state and local public health officials was issued by ASTHO in collaboration with NACCHO entitled “Effective Public Engagement: A Planning Guide.” The guide is available as Attachment C page 229 in the University of Nebraska Evaluation Report at the link: http://ppc.unl.edu/wp-content/uploads/2010/05/P5-Report-FINAL.pdf.

All links accessed April 2014.

Formal evaluation: Yes, the University of Nebraska Public Policy Center.

Links to project reports:

Several of the projects published individual reports, including:

Seattle/King County—Accessed April 2014
http://www.kingcounty.gov/healthservices/health/preparedness/pandemicflu.aspx

Minnesota—Accessed April 2014
http://www.health.state.mn.us/divs/idepc/ethics/ethics.pdf

Massachusetts—Accessed April 2014

Click to access 2.MACDCPANFLUFinalReport.pdf

Ohio—Accessed April 2014
http://www.cidrap.umn.edu/sites/default/files/public/php/452/452_socialdistancing_report.pdf

Links to published papers:

Health Care Decisions in Disaster: Engaging the Public on Medical Service Prioritization During a Severe Influenza Pandemic.
https://participatorymedicine.org/journal/evidence/case-studies/2010/12/14/health-care-decisions-in-disasters-engaging-the-public-on-medical-service-prioritization-during-a-severe-influenza-pandemic/
Accessed April 2014.

The Minnesota Pandemic Ethics Project: Sequenced, Robust Public Engagement Processes.

Accessed April 2014.

Link to evaluation report:

Click to access P5-Report-FINAL.pdf

Accessed April 2014.
Acknowledgements:
Approximately 1,000 citizens-at-large and stakeholder-citizens from 6 states participated in this project. See individual state reports for more detailed acknowledgments.

Case Abstract #8: Citizen Choices on the National Vaccine Plan

 

Performance period: 2009.

Background and question addressed:

Choosing the highest priority activities from a larger number of total potential activities for a national vaccine plan is a situation which creates many difficult choices between competing values of equal or near equal weight. Most often in the past, such decisions have been made with the benefit of advice from a sample of experts and stakeholders who hold particular values or have vested interests. Because a large and diverse number of citizens-at-large can be impartial and best reflect the public interest, the Public Engagement Project on the National Vaccine Plan was carried out to obtain an impartial, societal perspective on what should be the priority activities for the Plan.

Structure and process:

Four focus groups were held in the Chicago area in 2007 with 35 persons to identify public values that would be relevant for planning future public contributions to updating the national vaccine plan.

Subsequently, three public engagement sessions were conducted in St. Louis, Missouri, Columbus, Ohio, and Syracuse, New York. A total of approximately 250 citizens–diverse in race/ethnicity, gender, education,, and age—participated in day long, neutrally-facilitated sessions and small groups.

Lead organizations and sponsors:

The National Vaccine Program Office and 9 participating organizations, including:

• Centers for Disease Control and Prevention
• Columbus Public Health
• Data on the Spot
• National Vaccine Program, Department of Health and Human Services
• FOCUS St Louis
• FOCUS Greater Syracuse
• Oak Ridge Institute for Science and Education
• The Ohio State University College of Public Health
• One World Inc.
• University of Nebraska Public Policy Center

Facilitation and logistics: One World Inc. and Oak Ridge Institute for Science and Education.

Type of contributions by participants:

In plenary sessions and small groups, participants learned about vaccine-planning issues, identified and prioritized their values about vaccines, and applied those values as criteria to identify the most important amoung 12 areas of activity for the plan.

Outcome of the project:

Participants identified the values about vaccines that mattered most to them. They were: achieving equity, protecting the homeland, protecting the most vulnerable, having more education and awareness, and having more concern for safety. Using these values as criteria, the participants in at least two of the three cities identified the following as the highest priority areas: (1) improving monitoring of disease and vaccines, (2) making vaccines available and affordable for everyone, (3) maintaining high rates of vaccination of children, (4) assuring there is enough vaccine, and (5) improving vaccine safety.

Use/impact:

The final 2010 National Vaccine Plan identified 10 priority topics. Six of these ten were areas also prioritized by the citizens, and four were areas not considered as important by the public, including activities to promote vaccination globally, to increase the use of electronic medical records, and to target research on new vaccines. Overall, the public was most in favor of making sure existing vaccines are available and fully utilized for the US population, and several of the highest priority activities (top ten) in the plan were tied to improving use of existing vaccines in the US.

Conclusion for public engagement:

This project provided evidence that a diverse group of citizens-at-large can identify in day-long dialogue and deliberation sessions (1) what matters most to them about a topic area and (2) use these values as criteria to judge which proposed courses of action are best aligned with these values. This demonstrated that a sizable workload could be accomplished in a day-long dialogue and deliberation session.

The large number of parties consulted in the development of the plan made it difficult to parse out the influence of any particular group in influencing the choices made in the plan about the directions the plan would take. The narrative in the plan itself only briefly mentions the input from the public, and topic areas not mentioned by the public such as electronic medical records or topics voted of low importance by the public such as global activities were included in the final plan.

The usefulness of public engagement in planning work where the scope of knowledge of a subject matter needed to credibly identify and propose potential future actions is more limited than in policy work which is more dependent on values. The public generally does not possess subject matter expertise. To avoid this obstacle, the work of the public participants was focused on selecting priorities for the plan which is more dependent on values than expertise. However, even making this allowance, it was difficult to document the influence of the public’s advice in shaping the final plan. Six of the ten priorities selected may have emerged anyway without the public’s advice, and other objectives were either specifically not highly rated or added afterwards without the benefit of public input.

Formal evaluation: Yes, the University of Nebraska Public Policy Center.

Links to project reports:

Project and Evaluation Reports—Accessed April 2014.
http://ppc.unl.edu/wp-content/uploads/2009/08/nvp_dialogue_report_final.pdf
or at

www.hhs.gov/nvpo/vacc_plan/nvp_dialogue_report_final.pdf
The final 2010 National Vaccine Plan—Accessed April 2014.

www.hhs.gov/nvpo/vacc_plan/2010%20Plan/nationalvaccineplan.pdf
Final Implementation Plan for the 2010 National Vaccine Plan—Accessed April 2014.

www.hhs.gov/nvpo/vacc_plan/2010-2015%20Plan/implementationplan.pdf

The focus group report from the Oak Ridge Institute for Science and Education entitled “Public Values in Vaccine Planning” is available upon request to the corresponding author.

Link to evaluation report:

The complete evaluation report from the University of Nebraska is included as Appendix A in the final report. See above links for the project report.

Acknowledgements:

Approximately 250 citizens-at-large from Ohio, New York, and Missouri contributed their time and thinking to the difficult work of identifying critical values and then applying them to a set of proposed goals to identify the highest priority goals. Stakeholders from multiple organizations also contributed to the development of the plan through a separate project at the Institute of Medicine and through participation on the National Vaccine Advisory Committee.

Case Abstract # 9: The Public Engagement Project on the H1N1 Pandemic Influenza Vaccination Program

 

Performance period: 2009.

Background and question addressed:

When faced with gaps in data, uncertainty, and competing values, policy makers can look to the public and stakeholders to learn how they make tradeoffs and thereby help policy makers to make difficult decisions between competing approaches. The proposed plan for a mass vaccination program against H1N1 and whether or not to proceed full speed ahead in the face of uncertainty and despite the risks was such a situation. Recognizing this challenge, the Centers for Disease Control and Prevention conducted a series of public consultations in face-to-face meetings and online to better inform public health decisions on how best to prepare for the H1N1 mass vaccination program — take a “go-easy approach,” make a moderate effort to be prepared, or “go-full throttle.”

Structure and process:

Ten face to face meetings (one in each HHS region of the US) were conducted over a three week period in August 2009. Web dialogues took place in two sessions over two days each in late August and early September. A separate stakeholder meeting with representatives from affected sectors was convened in early September.

Lead organizations and sponsors:

The Centers for Disease Control and Prevention in collaboration with The Keystone Center, the University of Nebraska Public Policy Center, and WestEd.

Facilitation: The Keystone Center.

Type of contributions by participants

Participants spent time in small groups discussing the advantages and disadvantages of planning for three different target levels of preparedness, including “go-easy”, “go moderate”, and “go full-throttle” approaches. After the dialogue and deliberation sessions, the participants completed an electronic poll by voting their preference among the three levels of preparedness.

Outcome of the project:

A total of 1,095 persons participated at 13 events in 10 different geographic areas and in two online sessions. The majority of public meeting participants (52%) favored the moderate target level of preparedness for a mass vaccination program under the assumption that the severity of the pandemic would be similar to that of seasonal or annual influenza. Approximately one quarter of citizen participants favored either the “go-easy” target level (23%) or the full-throttle target level (25%) under the same assumption. In striking contrast, the majority of stakeholders favored a full-throttle approach (57%), with the next largest percentage favoring a moderate approach (40%), and only 3% favoring a go-easy approach.

Use/impact:

There is no evidence that the public’s advice about what level of mass vaccination readiness should be recommended by CDC to state and local health departments was ever adopted. In national guidance distributed in July and August before this project ended to grantees preparing for a mass vaccination program, CDC only indicated that the funds were to be used to “accelerate the speed of planning” and no guidance was issued about what the appropriate target level of readiness should be. No performance measures of preparedness were distributed and assessed.

Unofficially, many interpreted CDC’s expectation to be that grantees would move “full throttle” to achieve maximum readiness for large scale vaccination programs. In theory and in public, this is what officials claimed they were doing. In practice in the end, despite what may have been expectations to be maximally prepared to vaccinate a large percentage of the population, less than a quarter of all Americans were vaccinated, slightly higher among groups prioritized for vaccination. This suggests that while the public’s advice for a moderate effort was not accepted, it appears, in the end, it would have been the wisest course of action to avoid wasting effort and resources to best prepare to meet actual public demand for vaccination. It reveals the pitfalls of not aligning the political will and public policy with the public’s will in order to achieve large scale public good.

Conclusion for public engagement:

This project demonstrated that conducting public engagement activities in the midst of an emergency planning period can be of limited or no use for decision-making if decisions about critical questions have already been made or are made before the public consultation is completed. Also, this project revealed, as did the first two projects on vaccine prioritization, that stakeholders will assess decision options differently than citizens-at-large, at least publicly. Facing an uncertain future, stakeholders apparently reasoned it was better to be criticized for doing too much than doing too little whereas citizens reasoned a moderate approach was best. Failure to get better clarity for the decision being addressed by the public, the decision making process, and for the outcome product to be shaped by the consultations led to the limited influence which these public deliberations appeared to achieve in the decision making about the target level of preparedness most desirable at the state and local levels.

Formal evaluation: Yes, the University of Nebraska Public Policy Center.

Link to project report:

http://ppc.unl.edu/publications/public-engagement-project-on-the-h1n1-pandemic-influenza-vaccination-program/
Accessed April 2014.

A summary of the web dialogues is presented as Appendix D in the final report above.

Link to evaluation report:

http://ppc.unl.edu/wp-content/uploads/2009/10/Novel_H1N1_Vaccine_Evaluation.pdf
Accessed April 2014.

Acknowledgements:

The Keystone Center helped to organize this largest of all the nine projects carried out and did so in the shortest amount of time over a one month period during the pandemic.

More than 1,000 citizens-at-large contributed their time and careful thinking to provide advice about the difficult issue placed before them.

Summary and Discussion

The overall purpose of this initiative was to develop a feasible method of engaging the public that would better inform agency or sponsor decision making in the short term and build trust between the agency or sponsor and the public over the longer term. It succeeded in demonstrating feasibility and usefulness of the DPET model in the short term, but no ongoing infrastructure devoted to public engagement was created to achieve similar results over the longer term.

Overall, the DPET model was used on 8 occasions at the national level and on six occasions at the state or local levels. While there was variation in how the model was replicated, particularly at the state/local levels, its essential features such as focus on a clear pending decision, involvement of both citizens-at-large from different geographic areas, representation from the key stakeholder organizations, linkage with decision makers, presentation of essential information, and neutral facilitation of structured conversations were implemented across all of the national level projects except the Goals Project. The projects demonstrate that the DPET model is feasible and practical to carry out on a variety of difficult decisions across different agency sponsors. Since the projects carried out at the state/local levels were funded as grant programs, the CDC as the sponsoring agency had less control over the design and execution and greater variation ensued. In some areas, it was challenging from the outset to get clarity about what difficult public policy decision was the focus of the public consultation.

An important finding resulted from the inclusion of both citizens-at-large and stakeholders deliberating on the same questions. In several instances, the views were compatible, perhaps in part because the stakeholders had the benefit of hearing the views of the citizens prior to their own deliberations. However, in the instance of pandemic influenza rationing and on the issue of what target level of readiness to aim for in preparing to combat H1N1, the views of citizens and stakeholders diverged. Stakeholders appeared to view the public policy choices through the lens of their professional values or interests rather than from a framework centered on the broadest common good. If so, this finding highlights the importance of including citizens-at-large in public policy discussions to tap into the most disinterested perspective on public policy issues.

In terms of the consideration or influence achieved by the public’s advice, the results were mixed. For some projects such as those on vaccine prioritization, at risk populations, community control measures, vaccine safety, and in the Washington State and Minnesota projects, the public input was used to inform the decision making, and this was often acknowledged in the papers and project reports which emerged. Sometimes the input was useful in making the specific decision being considered. At other times, the input was helpful in revealing the extent of potential public support more than in shaping the decision itself.

For other projects, such as those on the National Vaccine Plan, there were multiple sources of input and it was not possible to isolate the contributions of the public. And there was only passing mention of the public input in the final report. For other projects such as those on the CDC Goals and the public consultation on H1N1 preparations, there is no evidence that the input was actually used. This occurred either because the agency’s original purpose in seeking the consultation was not carried out, making the advice irrelevant, or because the timing of the consultation precluded any real possibility that the public would have meaningful influence. Both of these negative outcomes may be preventable by seeking to obtain maximal clarity about the decision making process prior to any public engagement project. This is made more difficult in emergency situations where choices are evolving rapidly.

The effectiveness of the projects in building trust between the agencies or sponsors and the public in the longer term has not been evaluated. The evaluations that were carried out on several of the projects showed that most of the guiding principles were followed. It also demonstrated that they were successful in accomplishing such objectives as attracting citizens from diverse backgrounds to participate, improving the knowledge of the participants, and being perceived as high quality by the participants. The evaluations showed that there was an increase in trustin local government at the time the projects ended, and citizens reported they were more likely to participate in other types of engagement in the future. However, there was no follow up evaluation to measure this increased public-spiritedness over the longer term.

The public engagement projects begun as part of this initiative did not result in the creation of any permanent new infrastructure within the sponsoring agencies. A toolkit and planning guide were created to bring together resources and practical knowledge which state and local governments might use in future public consultations. The failure to build a stronger commitment within the sponsoring agencies to support public engagement so as to better inform decision making and to help maintain any possible improvements in public trust was disappointing given the overall degree of success achieved by the projects. The reasons for this failure appear to be numerous and complex, and are discussed at greater length in a companion “Lessons Learned” paper being published simultaneously with the present article in the Journal of Participatory Medicine.[1] It also includes possible approaches or recommendations to overcome the challenges going forward.

The greatest obstacle to productive and sustained use of public engagement lays not in lack of knowledge about how to engage the public as much as it does in a genuine will to do so. Sadly, there does not appear to be a culture in the United States which values public participation in public policy making. If anything, the culture, despite its democratic principles and aspirations, is dominated by elites.[7] These projects were not sufficient to overcome this resistance as embodied and reflected in the agency or sponsoring organizations even though there were demonstrable successes in achieving many of the stated objectives. Gridlock in making difficult, values-based public policy choices denies the benefits that sound policies can bring to the larger public. Greater recognition of the need for collaborative problem-solving and of the role which public engagement can play in this endeavor will be required to bring about more frequent use of decision-focused public engagement.

References

1. Bernier R. Lessons learned from implementing a multi-year, multi-project public engagement initiative to better inform governmental public health policy decisions. J Participat Med. 2014 May 21; 6:e8. ↩
2. Vaccines and issue of trust. Consumer Reports. August 2001: 17-21. ↩
3. Yankelovich D. Coming to Public Judgment: Making Democracy Work in a Complex World. Syracuse, NY: Syracuse University Press; 1991. ↩
4. The Keystone Center for Science and Public Policy. Final Summary Report and Proposal for the Vaccine Policy Analysis Collaborative (VPAC). The Wingspread Public Engagement Planning Group. September 2003. ↩
5. Institute of Medicine. Appendix E: History of public engagement at the national vaccine program office. In: Priorities for the National Vaccine Plan. Washington, DC: The National Academies Press; 2010. ↩
6. Bulling D, DeKraai M. Evaluation of a Multistate Public Engagement Process on Pandemic Influenza. J Participat Med 2014 Mar 8; 6: e5. ↩
7. Gilens M, Page B. Testing theories of american politics: elites, interest groups, and average citizens. Perspectives on Politics. In press. ↩

Copyright: © 2014 Roger Bernier and Caitlin Wills-Toker. Published here under license by The Journal of Participatory Medicine. Copyright for this article is retained by the author, with first publication rights granted to the Journal of Participatory Medicine. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution 3.0 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.