Abstract
Summary:
Objective: This study had two objectives: 1) to develop a feasible approach to quality indicator selection that incorporates the views of diverse stakeholders involved in tertiary/quaternary level child and adolescent mental health care, and 2) to identify a panel of quality indicators of high internal relevance for a specific site.
Methods: Stakeholders (N=18) (including patients/families, clinicians and administrators) were interviewed about their views on health care quality. Interview transcripts were grouped into themes. Quality indicators were reviewed and mapped to each theme.
Results: Four main themes were identified: 1) communication; 2) access to care; 3) evidence and outcomes; and 4) interprofessional care and engagement. Potential quality indicators of relevance in outpatient child and adolescent mental health care are outlined.
Conclusion: A “stakeholder-driven” approach, which invites discussion on health care quality from varying perspectives, presents a highly feasible and internally valid method of guiding quality indicators selection for a specific setting.
Keywords: Quality indicator, child psychiatry, mental health care.
Citation: Baribeau D, Wong J, Monga S, Pignatiello A, Ickowicz A. Selecting quality indicators in child and adolescent mental health care: a “stakeholder-driven” approach. J Participat Med. 2016 Mar 28; 8:e5.
Published: March 28, 2016.
Competing Interests: The authors have declared that no competing interests exist.
Introduction
The collection and monitoring of quality indicators has become a standard of practice in health care organizations. Some quality indicators (for example, rates of hospital-associated infections, mortality ratios, readmission rates) may be instituted by health ministries or governing bodies. [1][2] Increasingly, individual centers and organizations are also seeking to track various quality measures, to gain information for internal performance reviews and site-specific quality improvement. Despite a growing field of literature in this area, relatively few measures have been identified or studied in child and adolescent mental health.
In a recent review article on quality measures in child and adolescent mental health care, nine quality indicators were identified and recommended. [3] Evidence associating these specific metrics to improved outcomes was sparse, however; no indicators had randomized clinical trials (RCT) level evidence for example, and only 3 metrics had any evidence to support their use. [3] In the absence of stronger research, health administrators and clinician leaders must rely on expert opinion, or simply on feasibility, when it comes to indicator selection and monitoring. This is a “top-down” approach to ensuring health care quality risks that excludes relevant stakeholders, including front-line clinicians, patients and families. In a field where “quality” can have diverse meanings and perspectives, input from these sources is important.
Furthermore, many of these indicators have been developed for use in primary care settings, or for monitoring at a state/provincial level; therefore, they may not be relevant or actionable in specialized or consultative psychiatric care centers. The identification and validation of specific quality indicators for use in tertiary and quaternary child and adolescent mental health care settings is needed, to help ensure high quality service provision across a complex mental health system.
Accordingly, in the following study, we aimed to develop a low-resource, feasible method of quality indicator selection that incorporates the views of diverse stakeholders involved in child and adolescent mental health care in a tertiary outpatient setting. Our aim was to identify the most relevant aspects of health care quality worthy of monitoring, using a participatory “stakeholder-driven” approach, in order to guide the subsequent selection of quality indicators for our department. This research was completed as part of a Quality Improvement course, through the postgraduate residency program in Psychiatry, at the University of Toronto.
Methods
Participants
Stakeholders (n=18), including clinicians (n=7), patients/families (n=6) and administrators (n=5), were approached to participate in a quality improvement project. Clinicians included psychiatrists, psychologists, social workers, and resource navigators. Administrators included clinical and hospital directors.
Semi-Structured Interview and Analysis
A modified, quasi-experimental approach was used. Critical stakeholders were interviewed using a semi structured interview guide, and asked to identify the most important aspects of care that relate to health care quality at our center. Written notes and quotations were transcribed during the interview; no identifying information was collected.
We used a grounded approach to analyze the anonymous interview transcripts. We began with a process of open coding; a first pass through the transcripts revealed a number of tentative themes and categories. We then reviewed the transcripts and applied specific themes/ categories to each interview statement. Lastly, through a process akin to selective coding, we grouped individual themes and categories together, into major “core” categories, which captured several subthemes within. [4]
Quantitative Survey
A list of potentially important aspects of care (see Figure) was presented at the end of each interview. The items were generated by the authors via round table discussion. Stakeholders were instructed to select the three most important aspects of care pertaining to health care quality in outpatient child and adolescent mental health at our center, and had the option of adding additional items.
Indicator Selection
We surveyed the existing literature on quality indicators in child and adolescent mental health care. Indicators from American databases and agencies (including the National Quality Foundation, the National Committee for Quality Assurance, and the Agency for Health Research and Quality) have been extensively reviewed and synthesized by Zima and colleagues. [3] Additionally, we reviewed the Canadian Institute of Health Information (CIHI) quality indicator library, and selected mental health indicators applicable to child and adolescent mental health (for example, wait times, readmission rates). [1] Indicators were mapped to the core themes, and assessed by the authors on their relevance to primary vs tertiary care settings. Relevant indicators were compared by considering their validity, feasibility and importance at our center, in line with the recommendations of the US Agency for Healthcare Research and Quality. [5]
Results
Interview Data
Four overarching core themes emerged from our analysis: 1) communication, 2) access, 3) evidence and outcomes, and 4) inter-professional care and engagement. Several themes, primarily “communication” and “access,” encompassed additional subthemes within. Themes, subthemes, and select quotes are summarized in Table 1, and discussed in section 4.0. Saturation was reached after 18 interviews for the themes “communication” and “access.”
Table 1. Summary of core themes identified via interview, sub constructs, and examples of quotes (A: administrator, P/F: patient/ family, C: clinician).
Itemized Survey Responses
Results from the itemized survey are displayed in the figure. In summary, the distribution of items selected was in keeping with the themes reported via the semi-structured interviews. Looking at individual stakeholder groups, administrator responses were more heterogeneous across domains, while clinicians emphasized the importance of access to a team, and good bedside manner. Families, on the other hand, identified continuity of care with the same provider as being of high importance, much more so than other stakeholder groups and items.
Figure 1. Stakeholder responses to the question, “What are the three most important aspects of care for outpatient child mental health services as our center?”
X-axis items include: Clinicians who are caring, with good bedside manner; Detailed reports about the diagnosis and plan; Access to a team of professionals (psychologists, psychiatrists, nurses, social workers); Access to therapy; Short wait times for an appointment; Cost-efficiency; Continuity of care (seeing the same professional (doctors, nurses) for appointments); Degree of clinical expertise; Option of taking part in research studies and programs; Ease of being seen in this clinic, or other clinics in the hospital (accessible care). Y-axis: number of times each item was selected by various stakeholders.
Review and Selection of Potential Quality Indicators
The identified child and adolescent mental health care indicators, mapped to the core themes, are summarized in Table 2. A few additional indicators were proposed by the authors and added to this list (specifically, time to completion of written reports, family and patient satisfaction surveys, and staff engagement surveys), given their relevance to tertiary care, the themes identified, and our particular site. A further indicator (follow-up care for children prescribed ADHD medication within 30 days) was adapted, to include any psychotropic medication.
When health indicators were assessed for relevance in various care settings, five indicators were excluded from further consideration as they were found to be relevant to primary care only. These consisted of indicators that pertained to screening for mental health conditions (including screening for depression or developmental concerns using standardized tools), or contained the term “in primary care” in their description.
Of the remaining indicators, we selected three that we found to be sufficiently feasible, valid, and important at our center to be recommended to the department leaders: (1) time to completion of written reports, (2) wait times for first consultation, and (3) patient satisfaction survey.
Table 2. Quality indicators mapped to core themes.
Note: Italics indicate a site-specific indicator proposed by the authors. Other indicators included as summarized by Zima and colleagues. [[3]
Discussion
In this study, we used a stakeholder driven, quasi experimental approach to identify major themes of importance with respect to health care quality in an outpatient tertiary care setting for child and adolescent mental health services. We then used this information to help guide the selection of specific quality indicators of high relevance to the stakeholders at this site. Our main finding is that patients, families, clinicians and administrators described four overarching themes as being central to health care quality. These include: 1) communication, 2) access, 3) evidence and outcomes and 4) interprofessional care and engagement.
Communication
In review of interview transcripts, our first theme “communication” encompassed several subthemes within. Both patients and administrators emphasized clear communication around the procedural aspects of care (for example, who to contact, and where to go) as being important to quality, a sentiment that is perhaps reflective of the complex child mental health care system in the province of Ontario. [6] All groups commented on the necessity of clear communication around treatment plans, and the importance of empathic communication style. Notably, very few quality indicators recommended by national organizations capture these important aspects of care; therefore we added “standardized patient satisfaction surveys” as a metric.
On the other hand, numerous existing indicators assess written communication quality through chart audits, to ensure appropriate documentation of diagnostic criteria, and screening for risk symptoms. Through discussion with stakeholders, we felt that timely completion of written reports was also important in a tertiary setting, to facilitate communication with referring physicians, families, schools and other agencies. Notably, there is relatively little literature or specific guidelines with respect to timing of consult letters from specialists being sent to primary care providers. Most professional organizations highlight “timely” completion as a goal. [7] When surveyed, 69% of primary care providers wanted a letter from the specialist within 1 week of the appointment. [8]
Access
All stakeholders reported that timely and reliable access to care was a key factor when it came to quality. Many groups highlighted “wait times” as a core quality metric. Certain existing indicators capture aspects of accessing care and wait times; for example, “followup care [within 30 days] for children prescribed ADHD medication.” We felt that wait times for initial consultation was most important and feasible to monitor at our site, within the existing electronic system.
Clinicians also commented on wait times to access alternate/community programs and agencies: “If we can provide them with treatment, can that be in a reasonable length of time? If not, can we connect them to good resources in the community, in a timely manner.” Clinicians and administrators commented extensively on triaging procedures and scope of care as being related to the wait times issue: “Determining what we do and don’t see- our scope of care- [We need] good intake [and triage], -thorough intake, to screen in and screen out, so families aren’t disappointed.” The triaging process brought forward themes related to the identity of the department with respect to clinical scope.
Evidence and Outcomes
Families and administrators commented on the importance of improved outcomes, while clinicians highlighted the need for evidence-based interventions. We were not able to identify an indicator capturing “evidence and outcomes” of sufficient validity, feasibility and importance, to be implemented at our site, however. Routine screening was not felt to be of relevance to tertiary care, where in depth diagnostic assessments are the norm. The potential to use outcome questionnaires to survey overall functioning in a longitudinal fashion was considered as an option, but was not felt to be sufficiently feasible for implementation given the administrative burden of such surveys. The remaining two indicators (hospitalization rates for self-injury and readmission rates), were not considered sufficiently important at our center, where patients are primarily school aged children with neurodevelopmental, as well as mood and anxiety disorders, who have very low rates of hospitalization.
Interprofessional Care and Engagement
This theme highlighted the importance of interdisciplinary assessments and follow up for patients, as well as the importance of cohesiveness and collegiality amongst clinicians. These more subjective aspects of care are likely best captured by stakeholder surveys completed by clinicians and patients/ families. A staff engagement questionnaire is already employed annually at our center; therefore we selected a patient and family satisfaction survey as the indicator for this theme.
Limitations and Next Steps
The results of this study should be considered in light of certain limitations. First, the list of potential indicators was generated by review of databases and articles; it is possible that additional relevant indicators may exist that were not included. Next, saturation was reached for the themes of “communication” and “access” only; a larger sample of interviewees was not possible, but may have strengthened the robustness of the results. Next, while the results are highly specific and relevant to our site, the findings may not be generalizable to other centers. Moving forward, additional research is greatly needed to validate specific quality indicators in tertiary child and adolescent mental health care centers, in order to correlate these metrics to improved outcomes.
Conclusions
The above described approach, while relatively simple to conduct, is novel in the field of quality improvement, where traditionally, indicator selection and enforcement is implemented via a “top down” approach, by funding bodies, directors and ministries. Indicators developed for implementation across diverse settings and institutions run the risk of not capturing important aspects of quality in specific health care milieus. For individual health teams seeking to monitor quality internally, a stakeholder-driven approach is both feasible, and internally valid, as a method of capturing the most important aspects of care at a specific site. Therefore, while the indicators selected for our site may lack generalizability to other centers, the core themes identified can provide guidance around areas for potential indicator selection, particularly for other tertiary centers in child and adolescent mental health.
Acknowledgements
The authors would like to thank the patients, families, clinicians and administrators who participated in this quality improvement study. We thank Dr. Katherine Boydell for guidance around study design.
Ethical Considerations
This research was completed as part of a quality improvement initiative at the Hospital for Sick Children, in line with the guidelines of the department of Quality and Risk Management.
References
- Canadian Institute for Health Information. Health Indicators 2013. 2013. Available at: https://secure.cihi.ca/estore/productSeries.htm?locale=en&pc=PCC140. Accessed April 10, 2015. ↩
- Ontario Ministry of Health and Long-Term Care. The Excellent Care for All Act 2010. 2013. Available at: http://www.health.gov.on.ca/en/pro/programs/ecfa/legislation. Accessed May 18, 2015. ↩
- Zima BT, et al. National quality measures for child mental health care: background, progress, and next steps. Pediatrics. 2013; 131 Suppl 1: p. S38-49. ↩
- Creswell J. Qualitative Inquiry and Research Design. 3rd ed. Sage Publications; 2013. ↩
- Mangione-Smith R, Schiff J, Dougherty D. Identifying children’s health care quality measures for Medicaid and CHIP: an evidence-informed, publicly transparent expert process. Acad Pediatr. 2011; 11(3 Suppl): p. S11-21. ↩
- Ontario Ministry of Children and Youth Services. Moving on Mental Health. 2011. Available at: http://www.children.gov.on.ca/htdocs/English/topics/specialneeds/mentalhealth/moving-on-mental-health.aspx. Accessed March 3, 2015. ↩
- The College of Physicians and Surgeons of Ontario. Information Management: A system we can count on. 2006. Available at: http://www.health.gov.on.ca/transformation/providers/information/pdf/guide_bpd.pdf. Accessed May 18, 2015. ↩
- Tattersall MH, et al. Writing to referring doctors after a new patient consultation. What is wanted and what was contained in letters from one medical oncologist? Aust N Z J Med. 1995; 25(5): p. 479-82. ↩
- Wait Times Alliance. Wait Time Benchmarks for Psychiatric Illness. 2014. Available at: http://www.waittimealliance.ca/benchmarks/psychiatric-care. Accessed May 5, 2015.
Copyright: © 2016 Danielle Baribeau, Jennifer Wong, Suneeta Monga, Antonio Pignatiello, and Abel Ickowicz. Published here under license by The Journal of Participatory Medicine. Copyright for this article is retained by the authors, with first publication rights granted to the Journal of Participatory Medicine. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution 3.0 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.