Search all of the Society for Participatory Medicine website:Search


Background: In the Western world, governments and the public at large acknowledge the importance of strong patient advocacy groups as key players in health care. Patients are increasingly involved in the appraisal, evaluation, and decision making concerning health research projects and quality of care guidelines.
Objective: Previous work suggests that not all “issues that matter” to patient groups in practice are addressed in scientific studies. This study was designed to obtain a valid inventory of patient issues in order to close this gap.
Methods: Patient organizations were approached and interviews were held with selected members to elicit an inventory of issues that matter to patients. The issues inventory was then structured into a patient criteria list via open coding, clustering, and data synthesis. The criteria list was evaluated and adapted based on priority, sequence, and usability from a patient perspective in a series of focus group meetings.
Results: Six common patient appraisal criteria in health care and health research were identified: (1) Relevance, (2) Quality of Life, (3) Quality of Care, (4) Ethics and Safety, (5) Information & Communication, and (6) Involvement. The criteria “Quality of Life” and “Information & Communication” are presently being ignored in scientific studies. Only after basic values, such as respect, being recognized and acknowledged, and being accepted, are assured does a dialogue about relevant aspects for Quality of Life become possible. Participants in this study expect that the criteria will increase patient advocates’ awareness of the issues that matter and that using the patient criteria list will make their appraisals more complete, better structured, and more efficient.
Conclusion: Patient criteria for appraisal and evaluation of health research and quality of care have a priority structure that starts from fundamental values, proceeds to quality of life, continuing via quality of care and, finally reaching the right to speak. This structured set of patient criteria is expected to be of practical use, primarily to patient representatives but also to health researchers and other professionals.
Keywords: Appraisal, empowerment, patient involvement, shared decision making, quality of life, patient preference, issues, value, social inclusion, human dignity.
Citation: Teunissen GJ, Visse MA, de Boer WI, Abma TA. Structuring patient advocates’ appraisal and evaluation of health research and quality of care. J Participat Med. 2013 Apr 17; 5:e16.
Published: April 17, 2013.
Competing Interests: The authors prepared the article in the course of their employment at the Lung Foundation Netherlands in Amersfoort and VU University Medical Center in Amsterdam. The authors were in no way restricted by their employers in terms of research design, preparation time, or article content. No funds were received to conduct this study. The authors have no competing interests relevant to the content of this article. The opinions expressed herein are those of the authors.


Patients and their advocacy organizations desire to have a voice in the debate on health related subjects and want to have a say in the decision making process on health care and health research.[1]

Increasingly, patients take part in the debate on behalf of patient, disability, or elderly group. Patient advocates are often involved in providing helpful information to these groups of fellow patients and to assist in bringing patient issues into the debate. In research and health care policy, patient advocacy has gained broad acceptance.[2][3][4] Patient advocacy groups are considered to be important stakeholders in the improvement of health care research and policies.[5][6][7][8][9] They provide researchers and policymakers with an insiders’ perspective and on the meaning of living with a disease, and advocate important themes that research and policy should address. Patient involvement contributes to the quality of decision making processes and health practice, enhances the content, processes, and outcome of health care and increases legitimacy of decisions taken.[10][11][12][13][14]

There are several ways in which patients participate in research and quality of care. Patients may be included as partners in guideline development or research agenda setting.[6][7][15] When participating in research and policymaking, patient organizations generally work through persuasion efforts and exert their influence through the use of patient representatives.[16] Patient representatives face several challenges when they participate in an advisory board or committee. It is hard for them to articulate their feelings and opinions towards professionals due to a disparity in power.[17] Often committees use a voting system that can undermine the patient’s voice.[18] Furthermore, like anyone starting a new job who has doubts about what they can contribute, may not be clear about informal rules, and are not familiar with technical terms, inclusion of the patient’s perspective as a committee member is often hindered.[19] In order to enable patient involvement specific conditions are necessary. Occasionally, special procedures are designed to guarantee the genuine inclusion of patient perspectives in research.[20] Patient representatives need to know about values, needs, and issues of patients in their group in order to identify relevant issues among their own constituents, gather and oversee experience data, and formulate a relevant patient vision.[21]

Patient groups, therefore, not only struggle with the question of whether they are perceived as a fully equal participant in the decision making process, but also with how to usefully perform their role as a knowledgeable party. Given these challenges, patients lack instruments that will strengthen their role and improve their working relationships with professionals.[17]

An exploratory study in the Netherlands indicated that patient groups are being asked to participate more often, but that a knowledge gap remains between professionals’ issues and those introduced by patient representatives.[1] A review of international scientific literature on patients’ issues was carried out in an attempt to complete an “issues inventory.” [22] This revealed that such issues are better reflected in publications from government institutions and patient organizations than in scientific studies. It also demonstrated that the issues covered in the scientific literature do not accurately reflect what patients consider important in everyday practice.[1][22]

In earlier studies only a relatively small difference in “issues that matter” to patients was found between health research and quality of care domains. Apart from disease specific medical details, also little difference was found in patient issues between countries and in comparing different diseases.[22] The issues that matter to patients can be compared to Maslow’s theory of human needs.[23] Intertwined with these human needs are the values that drive human behavior. Patients’ issues are directly related to human needs and values. In order for patient representatives to be effective in their role, they must have an opportunitie to provide insight about these issues that really matter to patients. This leads to the following research questions:

  • What are the issues and values that matter to patients in quality of care and health research?
  • How can these issues and values be prioritized and structured into criteria for general use?


This qualitative study identifies the inventory of issues that matter to patients in order to enhance our understanding of these phenomena.[24] This was achieved through interviews with patient advocates, analysis and condensation of the issues into a list of criteria, and by assigning value for these criteria by conducting focus group meetings.

Defining Issues and Values

The term “issues” is used in this study for all claims and concerns that patients have concerning their disease and its consequences.[25] The term “value” reflects the ideals and goals for rules of good conduct. A classification of values with the central value “wisdom,” surrounded by some 43 other values, segmented into groups, called “dimensions” was developed around 1990 by Schwartz.[26]


Participant in interviews and focus groups were selected from four categories: (1) chronic illness, (2) disabled, (3) elderly, and (4) health care client councils. We included participant with one or more chronic illnesses or disabilities who are members of organizations or councils dedicated to their illness or disability (Table 1). Participants all spoke on behalf of their organization, participated in debates, and were providing advice on research and quality of care. Mentally ill or disabled people were indirectly included via spokespersons. All personal identifiers have been removed or anonymized.


This study was carried out in The Netherlands. Currently, there are an estimated 200 patient, disabled and elderly organizations and 560 client councils in institutionalized health care in this country.[27] A sample of 16 interview participants representing all of the four different organization types was considered to be sufficient to provide an initial indication of relevant issues in the practice of patient and client involvement.[28] The participants represent 14 different organizations of patients, disabled, and elderly people (see table 1).

Table 1. Interview participants

A further 52 respondents participated in three preparatory meetings and a series
of three focus groups, converting the initial issues inventory into a condensed
and structured set of criteria (see table 2).

After each focus group meeting, an analysis was performed. The criterion for data
saturation was that the last focus group meeting did not add any new insights.
After three focus groups, this criterion was met.

Table 2. Focus group participants

Interview Method

This study followed a responsive approach[25][29][30], in which the researcher acts as a facilitator to map stakeholders’ points of view. Issues were gathered using two types of interviews: semi-structured, in-depth face-to-face interviews and semi-structured telephone interviews for several chronically ill participants. Interviews typically lasted one hour face-to-face and 30 minutes by phone. The first two authors conducted the interviews.[31] The transcripts were sent back to the respondents for review.

Focus Group Method

Three preparatory meetings with patient advocates groups (18 total participants) were held by the first author. The issue clusters from the interviews were reviewed and participants were asked to express their views on issues, on linking them to criteria and to propose descriptions. The preparatory meeting participants were also invited to submit written remarks or suggestions.

The resulting concept criteria list was used for a series of focus group meetings, conducted in accordance with the Helsinki Declaration[32] and guidelines.[33][34] Only new participants, 34 in total, were admitted for each new focus group. At least one representative from each of the four group categories was present in each meeting. Each focus group lasted 2 to 3 hours and was conducted by the first two authors. Participants were first asked to review the inventory of issues, as condensed by the researchers into a concept criteria list, and associate these with values listed in the Schwartz classification.[26] Secondly, participants were asked to apply the concept criteria in two example cases (a research proposal and a care guideline development). Subsequently, subgroups were asked to inform the other groups whether they could benefit from the criteria in practice; whether they recognized them; whether they had suggestions for any additional issues and values or whether the concept criteria should be renamed; and whether they agreed with the sequence, priorities, and clustering the researchers had proposed. The researcher facilitated dialogue and asked clarifying questions. All focus groups were recorded and transcribed. Afterwards, a report of the process and outcome of the focus group was written and sent to the participants for review.

Interview Data Analysis Method

We used a thematic approach to analyze our data recorded in interview transcripts, using a content analysis focused on patient issues.[35] This analysis was done according to a combination of techniques: “thematic analysis,[36]” “meta aggregation,” and “data synthesis.[37]” Data gathering was conducted by manual extraction. Records were kept of the details associated with the extracted text and their source. After five transcripts, no new issue clusters emerged, implying data saturation.

Credibility and Reliability

Credibility “refers to the extent to which conclusions and inferences make sense to the people in the context studied.[38]” Safeguarding the credibility of our study also entailed conducting member-reviews after each interview and focus group meeting. The combination of methods (triangulation) added to the breadth of the study and helped to verify data. The research team discussed whether or not saturation was reached, and the process of data collection was ended when repetition of findings was found. To ensure inter-rater reliability, the first and second authors compared each others notes and recordings after the interviews and focus group meetings. Only consistent and common observations were used in transcripts, member reviews, and further analysis.


Patients’ Issues from Involvement Practice

The transcripts of face-to-face interviews were analyzed first. Subsequently, the transcripts of telephone interviews were analyzed. The first transcript was searched for issues encountered while participating in health care quality and health research. Some 420 text parts containing reference to a patient issue were selected and coded.

Logical relations between the issue text parts were found. For example, “gender” and “diversity” are logically related because gender is an element related to diversity. Such logical relations suggest that several issue texts may be combined into issue clusters. In order to cluster closely related or common issues in an inductive way, the coded text parts, representing all issues and activities mentioned in the first transcript, were subjected to open coding.[39]

The resulting issue clusters were used as a start, and led to additional clusters or to subdividing previously defined clusters. Coding of each next transcript started with the accumulated issue clusters from the preceeding transcripts. In total 33 issue clusters were derived.

Patient Criteria

Next we looked at ways to condense these issue clusters into groups. We used three steps in this analysis:

  1. First a detailed comparison was done between 35 issue clusters found in our previous literature study[22] and the 33 practice issue clusters found in this study. We observed that, in the literature, several “quality of life” and “fundamental human values” issues were not mentioned (share experience, recognition, independent voice, trust, be taken seriously, social participation, support, vulnerability, identity preserved). The authors believe the results from practice reflect the importance of these issues for patients.
  2. Then we looked at logical relations. In our previous exploratory study on patient issues in the Netherlands, three key terms were found: “Relevance”, “Right to Speak” and “Ethics and Safety.[1]” The key term “Right to Speak” has a logical relation with “Empowerment,” hence they were joined under the more general term “Involvement.” The range of “fundamental human values” issues is fully covered by the “Ethics and Safety” and “Quality of Life” key terms. Issues about conveying information to patients and communication with patients were both allocated to the key term “Information & Communication.” In this way we arrived at a core of six issue cluster groups, called criteria.
  3. Finally, we observed during the interviews that patients preferred a sequential order for specific kinds of issues within these clusters, which we think requires further research.

Establishing the Criteria Sequence

In order to validate the six criteria and to establish their relation and structure, homogenous and heterogenous (in terms of diseases) patient meetings were held.

First, three preparatory meetings were held within a homogenous group of 18 participants, all members of the lung patient advocates group at the Lung Fund Netherlands (LFN, formerly called Astma Fonds). In each meeting between 10 and 14 lung disease patients expressed their views on the above proposed six groups of issues clusters into patient criteria. They could change, add, and resequence them. This resulted in: “Relevance”, “Quality of Life,” “Quality of Care,” “Ethics and Safety,” “Information & Communication,” and “Involvement.”

In the debate during these meetings it became clear that the issue content itself was adequate but within the six criteria a new, value-related, sequential structure emerged. Participants were invited to reflect on this, verify the criteria description content and submit written comments about their view on criteria, values and sequential structure.

Validate the Criteria List Contents

Next, three heterogeneous focus group meetings were held, each with different participants. A total of 34 respondents participated in these meetings. They reviewed the concept “six criteria list,” leading to the following observations:

  • No new issue clusters emerged. This validates the issues content of the six criteria as obtained via the interviews among patient advocates and the preparatory meetings.
  • The descriptions of the six criteria were elaborated and questions were formulated to operationalize them as an appraisal tool.
  • The “Involvement” criterion was felt to be a prerequisite rather than an appraisal criterion.

Establishing a Sequential Structure

Then participants were asked to assess which patient values are involved in the appraisal of research proposals and quality of health care from a patient perspective. Starting with 14 values listed from the preparatory meetings and written responses received afterwards, no new values emerged in the first focus group meeting. The second focus group meeting identified one new value, namely “accessibility.” In the third focus group meeting no new values or other insights emerged.

Subsequently, participants were asked to describe the barriers they encountered during their involvement efforts.

Their first concern was “attitude” as an external factor limiting them in their activities in society.[40] Basic human values, like “being respected”, “recognized” and “accepted” and “not being excluded”, primarily determine the way participants think and act (Step I). “Trust” and feeling “safe” in spite of their “vulnerability” are important. Patients in their diversity want to do their “own appraisal” and expect to be treated as “equal” and with “fairness” and “dignity”. Hence, the more effectively fundamental human values are addressed the more the “quality of life” issues can become a concern.

Physical, social, and mental functioning constitutes a key factor in respondents’ lives. This means “well being,” “belonging,” “live a fulfilling life,” “self-actualization,” “compassion,” “freedom,” “autonomy,” and “self care” values are in order. They want to “be loved,” experience “hospitality,” feel good, and have work and a social environment with “family and friends” to experience a good quality of life (Step II).

After these fundamental and personal values are addressed, other subjects might become an issue. Access, freedom of choice, an agreed upon treatment plan, clear and complete patient information, effective communication on an equal basis between patient and care provider, an up to date medical file, and quick and adequate diagnosis are important when it comes to quality of care (Step III).[1][22]

Finally, a patient group has a right to speak when it comes to negotiation, advice, decision making, and guidance in quality of health care and health research matters. They consider “purpose and means,” and “what is right” and “responsible” as key issues. Empowered patients have found the strength and the ways to use experiental knowledge about their disease in an effective way. These are involvement issues (Step IV).

The participants thus indicated a clear preference to deal with issues in four sequential steps. The six patient criteria, descriptions and the related issue clusters with their sequential step numbers (I to IV) are shown in Table 3.

Table 3. Patient appraisal criteria list originating from involvement practice

Discussion and Conclusions


Because of practical funding and time limitations, some groups were under-represented in our study: patients with mental illness and patients belonging to various immigrant groups. Language and cultural issues, as well as non-membership of patient organizations, resulted in failure to include these groups.[5]

The differences in issues found between health research and health care domains, between diseases and between countries were small.[1][22] Hence, the geographical limitation of conducting the study in the Netherlands is expected to have little impact. This qualifies the current study as a first exploratory inventory of issues and values that matter to patients. Further study might provide more insight in, for example, disease specific issues and the need for special criteria lists.

The findings in this study underline the importance of fundamental human values grouped under the criterion “Quality of Life.[32][33][34]” Since the application of ethics and human rights in health research is not guaranteed, our findings could support patient organizations who attempt to monitor these important issues.

These patient criteria enable patient organizations to more clearly express concerns and share their experiential knowledge [41][42] with health care- and health research professionals. They may also help increase the skills of patient advocates.

Several European projects within the Innovative Medicines Initiative (IMI) are already [43] using the patient criteria list presented in this study.


This study resulted in a criteria structure linking issues and values that matter to patients. These appraisal and evaluation criteria will help patient advocates to become more aware of their patient groups’ issues, to more effectively engage in collective decision making, and to accomplish appraisal and evaluation in a more complete, structured, and efficient way.

Relevance, Quality of Life, Quality of Care, Ethics and Safety, Information & Communication, and Involvement constitute the six generally applicable appraisal criteria from a patient perspective. The preferred sequence to deal with these criteria was determined by patients.

We expect that this instrument will make patient involvement in research projects and development of guidelines more workable and effective. As a result the quality of research and guidelines will improve because the patient perspective is properly included.

The proposed criteria list is expected to be of practical use to many patient groups in many countries.


  1. Teunissen GJ, Abma TA. Derde partij: tussen droom en daad. Een verkennend onderzoek naar de patiëntenpartij en -criteria voor onderzoek, beleid en kwaliteit bij overheden en zorginstellingen. Tijdschrift voor Sociale gezondheidszorg. 2010; 88(4):182. (In Dutch)
  2. P1: Getting involved in research grant applications, Guidelines for members of the public, NHS – National Institute for Health Research, and P2: Peer reviewing research proposals. Guidelines for members of the public, DH Department of Health; 2009. Available at: Accessed November 2009.
  3. Kelson M . NICE committment to patient, carer and public involvement. In: Littlejohns P, Rawlins, M, eds, Patients, the Public and the Priorities in Healthcare. Milton Keynes, UK: Radcliffe Publishing; 2010.
  4. van Wersch A, Eccles M. Involvement of consumers in the development of evidence based clinical guidelines: practical experiences from the North of England evidence based guideline development programme. Quality in Health Care. 2001;10:10–16.
  5. Nierse C, Abma TA. Developing voice and empowerment: the first step towards a broad consultation in research agenda setting, Journal of Intellectual Disability Research. 2011; 55:411–421.
  6. Abma TA, Broerse JEW. Patient participation as dialogue : setting research agendas. Health Expect. 2010;13:160-173.
  7. O’Donnel M, Entwisle V. Consumer involvement in research projects: the activities of research funders. Health Policy. 2004;69 :229–238.
  8. Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database of Systematic Reviews 2006; Issue 3. Art. No. CD004563.
  9. Royle J, Oliver S. Consumer involvement in the health technology assessment program. International Journal of Technology Assessment in Health Care. 2004; 20(4):493–497.
  10. Epstein S. Patients groups and health movements. In: Hackett EJ, Amsterdamska O, Lynch M, Wajcman J, eds, The handbook of science and technology studies. Cambridge, MA: MIT Press, 2008:499-539.
  11. Baker A. Patient involvement in a professional body; reflections and commentary. J Health Organ Manag 2007; 21(4-5):460-9.
  12. Boote JD, Telford R and Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy. 2002; 61:213-36.
  13. Caron-Flinterman JF. A new voice in science: patient participation in decision making on biomedical research (thesis). Amsterdam: VU-Free University; 2005.
  14. World Health Organization. Everybody’s business: strengthening health systems to improve health outcomes. WHO’s framework for action. Geneva: WHO; 2007.
  15. Krahn M, Naglie G. The Next Step in Guideline Development. Incorporating Patient Preferences. J. American Medical Association. JAMA. 2008 Jul 23;300(4):436-8.
  16. Williamson C. Towards the Emancipation of Patients: Patients’ Experiences and the Patient Movement. Hobbs, UK: The Policy Press; 2010.
  17. de Wit MPT, Berlo SE, Aanerud GJ, Aletaha D, Bijlsma JW, Croucher L, et al. European League Against Rheumatism recommendations for the inclusion of patient representatives in scientific projects. Ann Rheum Dis 2011;70:722-726.
  18. Broerse JEW, van der Ham L , Veen S van, Pittens C, van Tulder M . Inventarisatie patiëntenparticipatie bij richtlijnontwikkeling, [Inventory of patient participation in guidance development]. Amsterdam: Athena Institute; October 2010.
  19. Hewlett S, Wit M de, Richards P, Quest E, Hughes R, Heiberg T, Kirwan J (2006). Patients and professionals as research partners: challenges, practicalities, and benefits. Arthritis & Rheumatism (Arthritis Care & Research). 2006 Aug15; 55(4):676–680.
  20. Elberse JE, Caron-Flinterman JF, Broerse JEW. Patient-expert partnerships in research: how to stimulate inclusion of patient perspectives. Health Expect. 2011;14(3):225-239.
  21. Schipper K. Patient Participation & Knowledge[Thesis]. Amsterdam: Vrije Universiteit faculteit VUmc, Medical Humanities; 2012.
  22. Teunissen GJ, Visse MA, de Boer WI, Abma TA. Patient issues in health research and quality of care: an inventory and data synthesis. Health Expect. 2011 Jul 20.
  23. Maslow A. , 2nd ed. New York: Harper and Row; 1970.
  24. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349-357.
  25. Meyer J. Using qualitative methods in health related action research. BMJ. 2000; 320:178.1.
  26. Schwartz SH. Universals in the content and structure of values: theoretical advances and empirical tests in 20 countries. Advances in Experimental Social Psychology. 1992;25:1-65.
  27. CG-Raad, LPGGz, NPCF. Platform VG [in Dutch; an agency that oversees patient, disabled, and elderly organizations in the Netherlands]. 2010. Available at: Accessed April 16, 2013.
  28. Guest G, Bunce A, Johnson L. How many interviews are enough? an experiment with data saturation and variability. Field Methods. 2006 Feb;18(1):59-82.
  29. Abma T, Widdershoven G. Responsieve methodologie. Interactief onderzoek in de praktijk. (Responsive methodology. Interactive research in practice.) The Hague: LEMMA; 2006.
  30. Guba E, Lincoln Y. Fourth Generation Evaluation. Thousand Oaks, CA: Sage Publications; 1989.
  31. Britten N. Qualitative interviews in medical research. BMJ. 1995 Jul 22;311(6999):251-3.
  32. World Medical Association. Declaration of Helsinki. Seoul, 2008.
  33. European Council. Additional Protocol to the Convention on Human Rights and Biomedicine on Biomedical Research. 2005.
  34. Council for International Organizations of Medical Sciences. International Ethical Guidelines for Biomedical Research Involving Human Subjects. 2002.
  35. Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research Chicago: Aldine Publishing; 1967.
  36. Thomas J, Harden TJA. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med res Methodol 2008;8:45.
  37. Lockwood C, ed. Reviewers’ Manual: 2008 edition. Adelaide, AU: Royal Adelaide Hospital; 2008.
  38. Greene JC. The construct(ion) of validity as argument. In: Chen HT, Donaldson SI, Mark MM, eds. Advancing validity in outcome evaluation: Theory and practice: New Directions for Evaluation, Number 130. San Francisco: Jossey-Bass; 2011:81-91.
  39. Berg BL & Lune H, eds. Qualitative Research Methods for the Social Sciences, 8th International Edition . Upper Saddle River, NJ: Pearson Education; 2011.
  40. WHO. International Classification of Functioning, Disability and Health: ICF. Geneva: WHO, 2001.
  41. Tong A, Lopez-Vargas P, Howell M, Phoon R, Johnson D, Campbell D, Walker RG, Craig JC (2012). Consumer involvement in topic and outcome selection in the development of clinical practice guidelines. Health Expect, 15(4):410-423.
  42. Kuper A, Lingard L, Levinson W. Critically appraising qualitative research. BMJ. 2008;337:687-692.
  43. Innovative Medicines Initiative. Available at: Accessed April 16, 2013.

Copyright: © 2013 G.J. (Truus) Teunissen, M.A. (Merel) Visse, W.I. (Pim) de Boer, T.A. (Tineke) Abma. Published here under license by The Journal of Participatory Medicine. Copyright for this article is retained by the author, with first publication rights granted to the Journal of Participatory Medicine. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution 3.0 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.