Abstract

Summary: Over the past decade, there have been many efforts to improve public engagement in health care. British Columbia (BC) began one such effort, called Patients as Partners, in 2008. Patients as Partners seeks to increase individuals’ engagement in their own health and health care, involve them in system redesign, and mobilize the community to improve health care outcomes.

As BC has adopted Patients as Partners, programs have tried to measure their success in implementing the philosophy. We describe the assessment strategies of four organizations using the measurement family outlined by Solberg and colleagues[1]: measurement for research, accountability, and improvement.

Engaging patients in developing measurement strategies has been slow and difficult. Nevertheless, because the process of measurement affects patient experience of care, patient input into measurement strategies is an important aspect of patient participation. Our experience in BC shows the need for a coordinated participatory approach that involves patients in measurement strategies; establishes consistent and efficient measurement strategies across systems; and encourages development of new measures of engagement.

As we work to increase public engagement in health care, it is important to consider how we will know if efforts to engage the public are successful. This article examines the implementation of the Patients as Partners philosophy in British Columbia (BC) and the simultaneous development of coherent and practical measurements to gauge its success.
Keywords: Patient-centered care, quality improvement, evaluation, patient engagement, public health.
Citation: McQuillen K, Davis CL, Ho K, McGowan P, Resin J, McEwan K, Kallstrom L, Rauscher C. Challenges in measuring patient participation: case studies from British Columbia’s Patients as Partners initiative. J Participat Med. 2013 Feb 27; 5:e10.
Published: February 27, 2013.
Competing Interests: The authors have declared that no competing interests exist.

Overview of Patients as Partners

In 2007, the primary health care charter of British Columbia (BC), developed by multiple community stakeholders, established Patients as Partners as the philosophy for primary care in the province.[2] The charter aims to achieve better health outcomes and improve the experience of professionals and patients at a sustainable cost.

In BC, we consider patients to be partners when “they are supported and encouraged to participate in their own health care and the decision-making about that care at the level they choose.[3]” To achieve this, we believe patients need opportunities “to participate in quality improvement and health care redesign in ongoing and sustainable ways.” We’ve sought to meet this goal of participation through a coordinated effort throughout BC.

The Ministry of Health promotes the Patients as Partners philosophy with the oversight of a committee that includes representatives from the Ministry of Health, regional health authorities, academic institutions, and non-governmental organizations. This committee set three priorities for Patients as Partners: (1) making patients partners in individual health care by supporting self-care, self-management, and partnership with professionals; (2) engaging patients as partners in redesigning primary care delivery by guaranteeing patient representation in the design of primary care; and (3) involving the community in health system redesign through work with residents and non-governmental organizations.

Each priority has a charter that describes its objectives, gives guidance for achieving goals, and proposes measures to monitor progress. As the Patients as Partners agenda has evolved in BC, the measurement strategy has been influenced by the need for measurement for multiple purposes: quality improvement, accountability and research.[1][4] Measurement for quality improvement (QI) seeks to inform ongoing efforts by providing immediate and continuous feedback to refine and improve the process or program. QI measures are kept to a minimum and are collected by those involved in the process. Accountability measures are traditionally used for program administration and evaluation. They are typically collected externally and have a “before and after” design. The delay in reporting makes them inadequate for ongoing QI purposes. Research measurement establishes new knowledge and is more exacting and expensive, with additional reviews and scrutiny required. Lloyd[4] notes that the critical first step in measurement is to determine the reasons for collecting and analyzing data. Solberg and colleagues advise caution about combining the three types of measures and our challenge has been to discover a way for the mixed purposes to co-exist.[1] Pronovost and Kazandjian[5] have proposed methodology to combine QI and research using modified clinical trial and case control methods and evaluation strategies are emerging that combine some aspects of quality improvement and evaluation.[6]

Case Studies in Measurement

We present case studies from the following four programs in BC whose leaders are members of the provincial Patients as Partners committee: (1) the Inter-Cultural Health Network (iCON), (2) Bounce Back, (3) the Patient Voices Network, and (4) the Practice Support Program (PSP).

These case studies were written by the programs’ creators. They illustrate how Patients as Partners preserves the unique strengths of different programs and organizations while pulling each toward a common set of provincial goals and measures. After a description of the program, each case study describes: (1) its approach to measurement, (2) patient involvement in development of measures, and (3) challenges in this early stage of the Patients as Partners’ work.

Inter-Cultural Online Health Network (iCON)

The iCON initiative partners with a variety of ethnic communities in developing evidence-based materials to better serve all patients in the diverse province of BC. So far, these efforts have included Chinese, Punjabi, and First Nations communities. Sponsored by the University of British Columbia’s Faculty of Medicine e-Health Strategy Office, iCON hosts interactive public education forums on chronic conditions led by a panel of policy makers, community organizations, health practitioners, and administrators. The initiative’s website[7] provides culturally appropriate health information and local resources in a variety of languages.

Approach to measurement.
The iCON initiative measures for improvement by tracking forum attendance and interactive web resource usage rates, and gathering participant evaluations both online and onsite. Our forums reach live audiences of between 500 and 900 and many online participants via webcasting. The team uses social media (Facebook, UBC eHealth, Twitter, eHealth Strategy) to promote the forum and measure participation. Google analytics tools track web-participation indicators, such as unique visitor visits, number of pages per visit, page popularity, duration of each visit, and longitudinal clustering of visits over time.

The initiative gathers participant feedback for purposes of quality improvement and impact measurement. Quality improvement data is collected using tools such as surveys, focus groups, interviews, and ongoing informal “feedback” via the iCON website’s Q&A, blogs, and Twitter followers. Quality improvement questions center on participant satisfaction with the ICON related tools such as the iCON website and events such as iCON’s culturally specific events.

Impact measurement is done by using surveys and focus groups. Participants complete surveys measuring levels of patient activation, e-health literacy, and lifestyle behaviors before and after the events. Specifically, post-event surveys ask questions about changes in knowledge and attitudes towards health related behaviors such as diet and exercise; perceived self-efficacy in making these lifestyle modifications; and, intention to change as a result of knowledge gained from events. Validated indices currently being used in surveys (pre and post) to measure impact include the Patient Activation Measure[8] and the eHEALS (eHealth Literacy Scale).[9] Focus groups are used to conduct needs assessment for culturally specific eHealth technologies such as web applications and text messaging.

Participant involvement in development of measures.
A significant component of our evaluation strategy uses surveys incorporating previously validated measures such as eHEALS[9] and Patient Activation Measures.[8] While community members were not involved in developing these indices, they play an active role in culturally adapting, translating, validating translations, and reviewing surveys and other data collection measures to ensure cultural appropriateness and relevance. Participants have also contributed to the development of an evaluation strategy by advising and helping identify critical areas of measurement as well as sociocultural considerations (importance of family systems and intergenerational, caregiver support, etc.) that have informed the development of culturally relevant evaluation activities as well as data collection tools (survey questions, focus group protocol). iCON liaises with South Asian and Chinese community partners through advisory groups to develop culturally specific, translated content for its events. Through the experiences and intimate knowledge of the health care providers, on the advisory panels, iCON strives to create an optimal self-management education experience for the community.

Measurement challenges.
A project of this scope encounters many measurement challenges:

• Paper- and web-based feedback mechanisms can present barriers to participants with low levels of literacy or physical constraints (eg, difficulty seeing written materials) and low technological literacy may limit access to online resources.
• Unfamiliarity with and cultural stigma against signing consent forms and providing formal feedback can hinder measurement, especially within certain high-barrier populations like First Nations and South Asian communities.
• Repeat participation makes it difficult to separate new and returning participant surveys.
• The large number of participants at live forum events makes it difficult to get adequate response rates while making sure that participants get all the help they need to complete the evaluation and that they understand the purpose of the evaluation.

Bounce Back

Bounce Back is a program administered by the Canadian Mental Health Association, BC Division. The underlying philosophy of Bounce Back is that, with guidance, individuals with mild to moderate depression can learn to manage their mood. The approach gives patients easy access to information on their condition and strategies to take control of their mental health and well-being. Patients in Bounce Back watch the DVD “Living Life to the Full” and complete a series of self-help workbooks. Participants then apply the skills they learned to different life areas. Bounce Back relies on a Participant (patient) Advisory Committee for input on program delivery.

Approach to measurement.
Bounce Back has developed process evaluation systems that provide ongoing feedback to improve quality. We collect referral data to assess referral targets and decide how to further promote the program with family physicians. Quantitative data is collected on the numbers of participants who complete the program, the number of those who begin the program but do not complete it, the number that decline service, and the number of referrals that do not meet program eligibility criteria. We define completion as those participants that complete both the pre and post program assessment for depressive and anxiety symptoms. Because Bounce Back is a self-dosing program (participants can work through all 16 modules, or fewer depending on their choice) some do not complete the modules and post measures. Measurement of sessions and workbooks completed gives us information on the issues worked on and the range and average of length of involvement in the program. These measures, reviewed monthly, help inform improvement strategies.

Bounce Back employs several accountability measures, including the Patient Health Questionnaire 9 (PHQ 9) measuring depressive symptoms; the Generalized Anxiety Disorder Questionnaire (GAD 7), measuring anxiety symptoms; the Quality of Life Enjoyment and Satisfaction Questionnaire; a physical health rating tool; the Patient Activation Measure (PAM); and a self-rated mood questionnaire.

Referring physicians complete a PHQ 9 prior to referral. We administer the PHQ 9, the GAD 7, the Quality of Life Enjoyment and Satisfaction Questionnaire and the physical health rating tool at baseline and post intervention. The mood rating questionnaire is completed each session and the PAM is completed at baseline and at 6 months following the baseline administration.

Scores on the PHQ9 and the GAD7 (N = 2896) indicate significant improvement over baseline on each outcome measure after program completion (paired sample t-test P < 0.001) with depressive and anxiety symptoms cut in half. Moreover, while the depression symptom scores of those who complete Bounce Back are normally distributed upon referral, after Bounce Back they are skewed markedly towards the minimal range. We administer a recently developed, three-question accountability measure before and after the program which measures understanding of how low mood and worry affects a person, knowledge of techniques to help oneself feel better, and confidence in managing mood and/or worry. Participant involvement in development of measures.
Bounce Back involves participants in measurement activities through the Participant Advisory Committee. This committee has been instrumental in reviewing and providing feedback for the newest accountability questions and a new multimodal participant survey.

Measurement challenges.
Because Bounce Back is essentially a self-dosing intervention in which participants decide how many sessions they need, sometimes participants are unavailable for end- of-intervention measurement. Thus, we cannot gather feedback on many participants’ experience with the program.

Patient Voices Network

The Patient Voices Network is administered by Impact BC for Patients as Partners, Ministry of Health. It is a mechanism to give patients, their families, caregivers, and community stakeholders a voice in health care system changes, with the main emphasis on primary health care. We expect that patients and caregivers can provide health system policy makers, administrators, and health care professionals with important information about how to best serve and partner with them in care.

The Patient Voices Network initiative has two main components: (1) a “virtual” network of BC patients who choose to receive information about primary health care in the province and may participate in surveys and become more involved through the activated network, and (2) an “activated” network which addresses two priorities: volunteer participation to “shape the system” through a variety of activities including speaking events, focus groups, advisory councils, and local and provincial workgroups; and community activation, engaging local organizations and the health system to support primary care.

Approach to measurement.
The Patient Voices Network uses a number of improvement measures to track progress, including the number of patients recruited, the patient and provider experience, the partnerships developed, and per-patient cost of engagement. Patient and provider experience measurement focuses on engagement during redesign in either the system or community. The survey includes how prepared patients were, how valuable contributions were, whether or not the aim of the engagement was met, and how likely providers and patients are to recommend Patient Voices Network to others. Monthly charts of the data gathered from engagement efforts and evaluation surveys of patients and providers help to change strategies when needed.

Participant involvement in development of measures.
Seven patients from across the province sit on the Patient Voices Network steering committee and help shape the charter and direction of the project, including measurement strategy. The survey was tested with Patient Voices Network members and provider partners in an iterative process until the current version was finalized.

Measurement challenges.
When developing measures, we use evidence-based questions from the literature whenever possible. Sometimes it is difficult to get stakeholders and patients to complete measurement requirements, like feedback forms, in a timely way, even though they are told this is mandatory at the start of each project. However, patients generally complete the evaluation surveys sooner than the professionals involved.

Practice Support Program

The BC Practice Support Program (PSP) is a program of the General Practice Services Committee which supports practice redesign efforts in primary care offices. In BC, primary care is provided through private, physician-owned offices staffed by a family physician and a Medical Office Assistant. PSP aims to improve the work-life of family physicians to enable them to provide better patient care. PSP provides several different modules, including mental health, end-of-life, and patient self-management with health literacy. From its initial focus on family physicians and their office practice, PSP has evolved to include specialist physicians. Most recently, we have begun to address the systems of care for specific clinical conditions, building new relationships with allied health care providers and community organizations.

Approach to measurement.
PSP measures for both program improvement and accountability to ensure program sustainability. Though PSP does not focus on research, it strongly promotes evidence-based practices and measures their translation into the day-to-day realities of physician practice. Measuring improvement in patient experience is a core component of the evaluation. To check quality improvement, measures are built into modules, discussed at practice team learning sessions, tested in action periods, and carried out through plan-do-study-act (PDSA) cycles. The teams tracking each module meet every two to three months to share their data and lessons learned.

In addition, through a standardized evaluation approach, the perspectives of family physicians, office staff, patients, and allied health professionals have been obtained via surveys. Surveys include measures of patient satisfaction with care and their experience, focusing on aspects of care that are covered in the modules, such as coordination of care with specialists and having enough time during visits to share their concerns. This survey data influences subsequent delivery of the modules, including determining measures that would be useful and baseline values for measures.

Patient involvement in measurement development.
Though involved mainly at the point of care delivery, patients have more recently begun advising in module development, learning sessions, and evaluations. Every new module includes patient or family representation to ensure that the module reflects patient values. During train-the-trainer sessions, patients and family members speak at the beginning of each session to ensure that patients are the focus of improvement. Several patients participate in each learning session, after they have been oriented to the session’s focus. Patients have contributed to survey design for modules. Patient Voices Network members (described above) provide input to the survey team. Thus, patients are able to contribute meaningfully to program development and delivery at all stages.

Measurement challenges.
Although measurement is seen as central to improvement, it has sometimes been difficult to reach consensus on the best measurement practices, especially when the desired data is unavailable. We often measure what we can get rather than what we want.

Ongoing measurement in a busy physician’s office has been challenging. A robust measurement system does not currently exist. One method of supporting physician office measurement is through deploying Regional Support Teams, who work with individual physician offices to troubleshoot measurement barriers. We expect that measurement capability will increase with the increase in use of electronic medical records across BC.

As more modules are adopted, physicians and the Regional Support Teams experience improvement and measurement fatigue. Though wanting to remain engaged, practitioners find it difficult to continually incorporate new ways of doing things into already busy practices. Data access and privacy also present issues and problems that need to be addressed in BC.

Other challenges include the difficulty in obtaining direct patient feedback for end-of-life care. As this would put too much strain on the patient, family members are involved instead. It is also very difficult to obtain patient input for the Child and Youth Mental Health module, even from teens, as it requires written parental permission. Instead we use parent representatives and young adults (over 19 years of age).

Current State and Moving Forward

The application of Patients as Partners principles is evolving in BC. While the culture is shifting to value patient experience and personal involvement in health care, practical methods and measurements are slow to develop. The Patients as Partners Advisory Committee has developed a provincial measurement strategy that includes a variety of instruments to gauge individual and community engagement and elements of system design (see Table 1).

Table 1.Provincial Measurement Strategy for Patients As Partners.

Despite this coordinated provincial measurement policy, the case studies highlight the diversity of approaches to measurement and the strategies supporting patients and health care providers to participate in measurement. They show that the uptake of provincial measures into the work of partner organizations is a work in progress. Recognizing this, the advisory committee created a measurement working group focused on measurement for improvement. The measurement group developed a common approach to viewing and describing measures as well as a catalog of measures, with a particular focus on measures that are important to patients. This catalog wiki[10] will further help to coordinate the measurement strategies of organizations integrating the Patients as Partners philosophies.

Discussion of Implications

Our experience instituting Patients as Partners in BC has led us to several conclusions about the best ways to implement a participatory medicine approach.

• Including patients and their interests through a defined participatory approach is central to this work. This includes involvement in the development, choice, evaluation and monitoring of measures and can help prevent issues of health literacy and cultural appropriateness.
• Coordinating measurement approaches and development throughout the system supports measurement for all purposes. This unified systemic approach can help maximize efficiency and information sharing between programs by providing more easily comparable data that measures program success.
• It is vital to pay attention to respondent, staff and system measurement burden. Web-based tools, such as www.howsyourhealth.org [11] may partially address this issue.
• A measurement strategy requires testing and development. The challenge is to support this development through partnerships that incorporate local flexibility while still aligning with provincial coordination. A measures template may be useful and one is under development in BC. Careful design and management of measurement will help to seamlessly integrate measurement coordination into an existing health care system.

Conclusion

In BC, instituting Patients as Partners has given us an opportunity to closely examine the process of integrating the principles of participatory medicine into a health care system. This knowledge will inform the further development of the Patients as Partners work. Our case studies have highlighted the challenge of measuring effectively to gauge success while keeping the burden of measurement from jeopardizing success. As the field of participatory medicine grows, greater understanding of how to measure success at a variety of levels will continue to inform the engagement of patients and the community in the design and delivery of care.

References

1. Solberg LI, Mosser G, McDonald S. The three faces of performance measurement: Improvement, accountability and research. Jt Comm J Qual Improv. 1997;23(3):135-147. ↩
2. British Columbia Ministry of Health. Primary Health Care Charter: A Collaborative Approach. 2007. Available at: http://www.health.gov.bc.ca/library/publications/year/2007/phc_charter.pdf. Accessed February 8, 2013. ↩
3. British Columbia Ministry of Health. Developing a shared vision for Patients as Partners: an essential element of the British Columbia Primary Health Care. BC Primary Health Care Charter: Patients as Partners Workshop, Vancouver, BC, July 14-15, 2008. ↩
4. Lloyd RL. Navigating the turbulent sea of data: the quality measurement journey. Clin Perinatol. 2010;37:101-122. ↩
5. Pronovost PJ, Kazandjian VA. A new learning environment: combining clinical research and quality improvement. J Eva Clin Pract 1999;5:33-40. ↩
6. Evaluation of place-based approaches: Questions for further research. Government of Canada, 2011. Available at: http://www.horizons.gc.ca/doclib/2011_0074_EvaluationPB_e.pdf. Accessed February 8, 2013. ↩
7. Intercultural Online Health Network. Available at: www.iconproject.org. Accessed February 8, 2013. ↩
8. Hibbard J, Stockard J, Mahoney ER, Tusler M. Development of the patient activation measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004;39(4):1005-1026 ↩
9. Norman, C. & Skinner, HA. eHEALS:The eHealth literacy scale. J Med Internet Res 2006;8(4):e27. ↩
10. ImpactBC Measures catalog. Available at: http://measures.impactbc.ca. Accessed February 8, 2013. ↩
11. How’s Your Health. Available at: www.howsyourhealth.org. Accessed February 8, 2013. ↩

Copyright: © 2013 Kelly McQuillen, Connie Davis, Kendall Ho, Patrick McGowan, Joyce Resin, Kimberley McEwan, Liza Kallstrom, Chris Rauscher. Published here under license by The Journal of Participatory Medicine. Copyright for this article is retained by the authors, with first publication rights granted to the Journal of Participatory Medicine. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution 3.0 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.