example of quantitative research about mental health

Quantitative needs assessment tools for people with mental health problems: a systematic scoping review

• Open access
• Published: 12 March 2024

Cite this article

You have full access to this open access article

• Irena Makivić   ORCID: orcid.org/0000-0003-2748-5522 1 ,
• Anja Kragelj 1 &
• Antonio Lasalvia   ORCID: orcid.org/0000-0001-9963-6081 2  

223 Accesses

Explore all metrics

Needs assessment in mental health is a complex and multifaceted process that involves different steps, from assessing mental health needs at the population or individual level to assessing the different needs of individuals or groups of people. This review focuses on quantitative needs assessment tools for people with mental health problems. Our aim was to find all possible tools that can be used to assess different needs within different populations, according to their diverse uses. A comprehensive literature search with the Boolean operators “Mental health” AND “Needs assessment” was conducted in the PubMed and PsychINFO electronic databases. The search was performed with the inclusion of all results without time or other limits. Only papers addressing quantitative studies on needs assessment in people with mental health problems were included. Additional articles were added through a review of previous review articles that focused on a narrower range of such needs and their assessment. Twenty-nine different need-assessment tools specifically designed for people with mental health problems were found. Some tools can only be used by professionals, some by patients, some even by caregivers, or a combination of all three. Within each recognized tool, there are different fields of needs, so they can be used for different purposes within the needs assessment process, according to the final research or clinical aims. The added value of this review is that the retrieved tools can be used for assessment at the individual level, research purposes or evaluation at the outcome level. Therefore, best needs assessment tool can be chosen based on the specific goals or focus of the related needs assessment.

Similar content being viewed by others

How to use and assess qualitative research methods

Loraine Busetto, Wolfgang Wick & Christoph Gumbinger

STOPP/START criteria for potentially inappropriate prescribing in older people: version 3

Denis O’Mahony, Antonio Cherubini, … Denis Curtin

The Person-Centred Nursing Framework

Avoid common mistakes on your manuscript.

Mental disorders are the largest contributor to the disease burden in Europe (Wykes et al., 2021 ), and mortality related to such conditions increases the overall economic burden (McGorry & Hamilton, 2016 ). Mental disorders affect various life domains, from physical health to daily living, friends, family situations, and education, and are associated with greater unemployment and economic problems (Wykes et al., 2021 ).

In order to plan and carry out successful mental health care, it is necessary to have a good mental health information system that also includes data about related needs (Wykes et al., 2021 ). When a need is identified, an action can be (re)organized to address it. Such action, based on the needs identified by the affected individuals, professionals or society, results in either satisfaction or dissatisfaction if the needs continue to be present (Endacott, 1997 ). Assessing needs might also be used to assess the adequacy and prioritization of mental health services at the population level (Ashaye et al., 2003 ; Hamid et al., 2009 ) as well as for the evaluation of mental health care (Hamid et al., 2009 ).

When considering mental health, a need represents a gap between what is and what should be (Witkin & Altschuld, 1995 ), and any changes that are made to the system should thus work to reduce this gap. There are various definitions of both “need” and “assessment” (Royse & Drude, 1982 ). Kahn (1969) considered needs from a social perspective to represent what someone requires in a broader bio-psycho-social context to be able to fully and productively participate in a social process (Royse & Drude, 1982 ). Brewin conceptualised needs (Lesage, 2017 ) as assessing what kind of social disability an individual has for professionals to be able to use an adequate model of care. Disability in this context is the result of interactions between people and the environment, and thus a disability can be seen as a lack of appropriate care models in relation to recognized needs. The concept of “need” in mental health care may be defined according to different points of view: a “normative need” is defined by professionals, while a “felt need” is what people with mental health problems experience and ask to be met (Endacott, 1997 ). What patients request and what they really need may differ, as they can only get what is available and provided at the system level, and what is the most beneficial for them in the current situation. Moreover, what they ask for is not always feasible. However, according to Bradshaw, what an individual requests is important and should be considered as felt needs (Endacott, 1997 ). Bearing in mind Maslow’s hierarchy of needs, only a combination of assessments from different points of view can provide a comprehensive needs assessment: needs assessed at the individual level from service users, their family members, caregivers, practitioners, and other professionals (Endacott, 1997 ). Indicators of needs at the individual level include functioning on different levels, symptoms, diagnoses, quality of life and, access to services (Aoun et al., 2004 ). Patient-centredness is vital to ensure the highest quality of care through monitoring performance (Kilbourne et al., 2018 ). Taking into account the patients’ perspective is also important to assess needs correctly, since such an assessment is more than just the professionals’ perception. An assessment of needs, as Thornicroft ( 1991 ) pointed out, provides care in the community with an emphasis on the provider-user relationship as a key component through which effective care is organized (Carter et al., 1995 ). According to Slade ( 1994 ), the concept of a need in mental health has no single correct definition, but it should rather be seen s a “socially-negotiated concept” (Thornicroft & Slade, 2002 ). Additionally, needs have to be assessed through the bio-psycho-social model (Makivić & Klemenc-Ketiš, 2022 ), including not just medical needs but also a wide array of social needs.

Initially, the assessment of needs (Balacki, 1988 ) in the community was seen as an approach using different forms of analysis to gain insights into the use of services, characteristics of people, incidence and prevalence rates and indicators to recognize crucial determinants that lead to the worsening of mental health. The assessment of mental health needs in Western societies began in 1775 with the analysis of public health data contained in the case registers (Royse & Drude, 1982 ). In the mid-1970s, with the beginning of the transition to care for mental health in the community (and the launch of community mental health service organizations), needs assessment was required within the evaluation process to help meet the patients’ needs. Needs assessment also represents a crucial part of mental health planning (Royse & Drude, 1982 ), where different needs must be considered, especially those felt by individuals. At the end of seventies, Kimmel pointed out that this area of needs assessment had no systematic procedures (Royse & Drude, 1982 ). However, several mental health needs assessment tools have been developed over the last thirty years.

The MRC Needs for Care Assessment (NFCAS) (by Brewin, 1987) was the first attempt to introduce a standardized assessment of the needs of the severely mentally ill (Lesage, 2017 ). Subsequently, a reduced version of the instrument applicable to common mental disorders was developed – i.e., the Needs for Care Assessment Schedule-Community version (NFCAS-C) (Bebbington et al., 1996 ). The shortened version of NFCAS was the Cardinal Needs Schedule (CNS), which is used to assess needs to address them with appropriate interventions (Marshall et al., 1995 ). Later the self-administered Perceived Needs for Care Questionnaire (PNCQ) was developed for use at the population level (Meadows et al., 2000 ), while in 1995 the Camberwell Assessment of Need (CAN) (Phelan et al., 1995 ) was published. After this time the focus shifted more to people-centred approaches, and therefore the assessment of needs also moved beyond psychiatric symptomatology to bring in “consumers”, i.e. patients and their caregivers. Other scales have also been used as needs assessment tools, such as the HoNOS scale (Joska & Flisher, 2005 ) which was designed to evaluate the clinical and social outcomes of mental health care.

Needs assessment is not always a clear and straightforward process with one approach and one goal. Therefore, different tools and approaches may be used to assess needs from different perspectives at different levels and with the help of different tools. The problem with using different techniques is that there is a lack of comparability and a consequent danger of not using the needs assessment outcome data as intended (Stewart, 1979 ); thus, it is important to have a good overview of the available tools.

To the best of our knowledge, only six reviews on needs assessment in people with mental health problems have been published to date (Davies et al., 2018 , 2019 ; Dobrzyńska et al., 2008b ; Joska & Flisher, 2005 ; Keulen-de Vos & Schepers, 2016 ; Lasalvia et al., 2000b ). Four additional reviews focused on the general needs or general health needs of people without mental health problems (Asadi-Lari & Gray, 2005 ; Carvacho et al., 2021 ; Lasalvia et al., 2000a ; Ravaghi et al., 2023 ), which was not focus group of our review. Finally, another article was considered inadequate for this study’s purposes, as it was published in Polish (as the one above) and is not a review paper (Dobrzyńska et al., 2008a ). None of the reviews published thus far have focused on the different assessment tools available for assessing the needs of people with different mental disorders. To date, no study has attempted to review all the available published studies on the various needs assessment processes to systematize the topic. The reviews mentioned above deal with only one specific population (patients with first-episode psychosis; forensic patients), or with specific needs (need for mental health services, supportive care needs, or individual needs for care). Thus, this study aimed to review all studies addressing needs assessment tools specifically designed for people with mental health problems, regardless of their diagnoses. The added value of this study is especially because of its wholeness in presenting different tools that can be used on different populations and by different groups. Thus this study may serve as a framework for starting different needs-assessment processes.

Search strategy

A comprehensive literature search using the Boolean operators “Mental health” AND “Needs assessment” was conducted in electronic bibliographic databases PubMed [Needs Assessment (Mesh Terms) AND Mental Health (Mesh Terms); Mental Health (Title/Abstract) AND Needs assessment (Title/Abstract);] and PsychINFO [Needs assessment AND Mental health in keywords; Needs assessment AND Mental health in Title; Needs assessment AND Mental health in Abstract]. Searching was carried out with the inclusion of all results without time or other limits in August 2021. The search strategy was based on the needs from a clinical context as well as some research priorities in the field of mental health. After the first systematic search we collected additional papers with an overview of six review articles (Davies et al., 2018 , 2019 ; Dobrzyńska et al., 2008b ; Joska & Flisher, 2005 ; Keulen-de Vos & Schepers, 2016 ; Lasalvia et al., 2000b ) and their results, and by searching PubMed within all connected articles. This was important since keywords changed over all this broad timeframe.

Inclusion and exclusion criteria

Our research exclusively focused on quantitative studies. We thus excluded all theoretical/conceptual articles, editorials, books, book commentaries or dissertations. Studies assessing the needs of patients with dementia and groups of people with physical and psychological disabilities were also excluded. We did not include papers related to 1) only general health (care), 2) other needs of the general population, 3) screening, prevalence, general diagnostic tools, and 4) tools for assessing caregivers’ needs. All those steps were done comprehensively by two researchers (IM, AK) independently. When there was a disagreement on the inclusion or exclusion of an article, both researchers looked at it again before reaching a consensus. We then manually added all relevant articles that could have been missed during the electronic search. We added articles that were cited within or were related with all the six mentioned reviews, but were not yet retrieved in the first search. These review articles were not included in the final number of all the articles examined in this study with the aim of exploring the different tools used for needs assessment of people with mental health problems. The aim of this process is to first obtain an overview of all the tools available, as this will make it possible to better use them within clinical settings, as well as for research and development purposes in order to plan a system or intervention that addresses the recognized needs (Fig.  1 ).

Concept of patient-centred care based on needs

Scoping studies, as Arksey and O'Malley ( 2005 ) mentioned, follow five steps, which we also took into consideration. First (step one) we identified the research question, which was “What are all different needs assessment tools that have been used in the population of people with mental health problems within different studies”. We then identified the relevant studies within recognised databases, as well as manually searching and adding the relevant articles (step two). We selected the appropriate studies (step three) as described within the search strategy process, with all inclusion and exclusion criteria. Finally, we presented the results (step four) in the chart flow in Fig.  2 , and Tables  1 , 2 and 3 , which corresponds to the concept of patient-centred care based on needs (Fig.  1 ). Because our focus was on different tools, we prepared the tables accordingly. There was no other relevant information in the original 242 articles to be presented at this occasion, other than those about the usage of different needs assessment tools, as this was the goal of the scoping review. The presentation of the results is based on the use of all recognized needs assessment tools, since geographical studies have been presented elsewhere (Makivić & Kragelj, 2023 ).

Research process within the databases

The analysis was multi-structured to provide an overview of all the recognized tools and the related time trends, country use and population of the most frequently used assessment methods.

The study selection process is shown in Fig.  2 . PubMed provided 578 records within the Mesh search and 537 within the title/abstract search, with after duplicates were removed this gave 1,090 results. Searching in PsychINFO provided 650 results from a search within the Abstract, 232 within Keywords and 1450 within Title; after combining these and removing duplicates, a total of 1,548 results were obtained.

The first selection was made within the final database (n = 2,638) by reading the abstracts and excluding all studies covering topics not relevant for this review. After this was completed, 166 articles remained. These were reviews and research articles covering the needs assessment of people with mental disorders (MD). After this, we eliminated review articles (n = 6) and used them for additional search to manually add all relevant articles that could have been missed during the electronic search, mainly because of the use of different keywords. Specifically, we added the articles that were cited within or were related to all the six mentioned reviews, but were not found in the first search (n = 82). After this process, a total of 242 articles were included in the final review.

Most studies addressing needs assessment tools retrieved with both electronic and manual searches were published in English (n = 231), although some were published in German (n = 3), Spanish (n = 3), and Italian (n = 2). Only one article each was published in Dutch, French and Turkish. Regarding the geographical distribution, most studies were published from European groups (n = 163), while 43 studies were conducted in America, 22 in Australia or New Zealand, 11 in Asia and only three in Africa. Some of the studies were published in collaboration among researchers from different countries. Regarding the publication period, the first studies on this issue were published in 1978, 52.9% of the studies were published from 2000 to 2012, and 66.1% had been published further by 2016.

Through the search performed in this study we found 29 different needs assessment tools, as shown in Table  1 in alphabetical order. We have made and additional search in order to find original sources and the information about the validation. Original sources for each of the recognized tools are listed in Supplementary information ( SI 1 ). Some tools, additional to those 29, were developed for the purposes of a single research study and its specific aims and the information about the validation were not available (n = 11), and thus we eliminated those tools at this point, although they will later be presented elsewhere in another study.

The retrieved tools and their respective constructs of need are presented in Table  2 . The various needs assessment tools are listed in alphabetical order. The tools are presented with regard to (1) who can answer the scale, (2) who the target population is, and (3) the domains addressed. Table 2 provides information on the various needs assessment tools, listed in alphabetical order. The tools are presented with regard to (1) who can answer the scale, (2) who the target population is, and (3) the domains addressed.

Service needs (Hamid et al., 2009 ) are defined as care requirements for prevention, treatment and rehabilitation. These needs can either be assessed by waiting lists or by only asking a simple question (e.g. “Do you think that you require any professional mental health services?”) along with the screening for mental and physical health problems (Yu et al., 2019 ) or social problems, with the help of the tools listed below. Moreover, there are different bio-psycho-social needs that are related to various mental health, physical health, and quality of life factors, as well as personal interests or abilities and social factors (Keulen-de Vos & Schepers, 2016 ), and these can be measured for different purposes. Social needs can be assessed by tools such as the Social Behavioral Schedule or REHAB Schedules, and therefore the need for rehabilitation can also be assessed (Hamid et al., 2009 ) using the comprehensive tools mentioned in our review.

Most of the needs assessment tools were self-completed by the patients (n = 85), completed by professionals (n = 41), or by combination of both (n = 78). Some tools were also completed by the patients and their caregivers (n = 12) or by the patients, caregivers, and professionals at the same time (n = 12). There were few studies where the researchers completed the needs assessment tool (n = 5). The majority of the tools were developed for assessing needs in an adult population with mental health problems (n = 193), either with severe mental disorders or with some other mental health diagnosis. Seventeen studies focused on an elderly population with mental health problems, and six on children with mental health problems. Some needs assessment tools for specific populations were found, such as tools for assessing the needs of forensic patients with mental health problems (n = 18), homeless people and migrants with a mental health diagnosis (n = 4), and mothers or pregnant women with a severe mental disorder (n = 1). In some studies, there was a combination of all these different populations and even people without a diagnosis, which we assigned to each of the mentioned groups.

In the second Supplementary information ( SI 2 ) there are reported the studies found in the literature search that used recognized needs assessment tools (n = 227). In this presentation some of the studies are not presented, namely those without validated tools (n = 11) as already mentioned and all articles using mentioned three different models (n = 4). In some studies, more tools have been used and in this case the study is counted within each tool in the total number of studies. Among the different needs-assessment tool, the CAN is mentioned as the most frequently used scale and, to the best of our knowledge, it has the highest number of different versions. The tools are presented based on their frequency of recognized use within this scoping review, from the most frequent to the least.

The recognized tools can be used in different contexts. Table 3 , groups the needs assessment tools according to their use at the care, research, and system levels.

This scoping review addressed all the published needs assessment tools specifically designed for use in mental health field. Nevertheless, some of the reviewed tools had also been used on the populations without a mental health diagnosis (Carvacho et al., 2021 ). Overall, we found twenty-nine different tools measuring needs in various mental health populations. The list of authors of the originally developed scales mentioned below are provided in the Supplementary information ( SI 1 ).

The reviewed literature highlights that the majority of needs assessment tools have been developed and used in Europe as the adoption of a community psychiatry model is relatively more widespread in this region than in other world regions; some tools, however, have been also used in America, Australia, and New Zealand.

Some scales had been developed with the aim to simplify or shorten previously published needs assessment tools, such as the Camberwell Assessment of Need (CAN) derived from the MRC Needs for Care Assessment Schedule. Similarly, the Difficulties and Needs Self-Assessment Tool was derived from the CAN, where some items are identical, some are a combination of several items of the CAN and some were added as new ones (on work, public places, family and friendship). Some tools, like the Montreal Assessment of Needs Questionnaire, were also developed from the CAN and had different aims, like enhancing data variability to broaden outcome measures for service planning, or simply because the organization of the related system is different and other tools are more appropriate. On the other hand, some tools are based on the CAN, but have been designed for use on a larger scale at the population level, like the Needs Assessment Scale. While most of the tools are used within health care services, the Resident Assessment Instrument Mental Health is a tool developed to support a seamless approach to person-centred health and social care. Some of the tools can also be used outside of the mental health field – such as the Child and Adolescent Needs and Strengths, which can be used in juvenile justice, intervention applications and child welfare – and the abovementioned CAN and others.

There are slightly different ideas regarding the needs and concepts about measuring needs. Many tools include a combination of needs assessed from different perspectives, such as the Bangor Assessment of Need Profile and the CAN. In some tools, like the Community Placement Questionnaire, it is predicted that various people rate the situation for one patient to eliminate any inaccuracies. On the other hand, some tools presented here, like the Self-Sufficiency Matrix, measure needs indirectly through self-sufficiency. When there is higher self-sufficiency for a certain life domain then there is less need presented for this area. Some tools, like Services Needed, Available, Planned, Offered, are complicated to use, since they include an investigation method with the review of the tool and assessment of the service use after the needs have been recognized. But this can be a good approach for the evaluation of the performance of community mental health centres about meeting the needs of their patients. Although we must bear in mind that such a tool is not directly transferable to every community mental health centre, as this depends on how each system is organized.

Needs can be evaluated according to different points of view, from patients themselves and their caregivers, as well as professionals. Studies show there are different outcomes based on the assessor (Lasalvia et al., 2000a , b , c ; Macpherson et al., 2003 ), and that professionals may see the needs differently to the users. Therefore, it is important not only what the tool is being used, but also who can complete it. Therefore, the most useful tools are the ones that can be used by various different people, so that the needs are assessed (also) from the patients’ standpoints (Larson et al., 2001 ).

Although the CAN is the most widely used tool, the research shows that sometimes there is not a very high agreement between staff and patients about needs, as was also found with the Health of the Nation Outcome Scales (HoNOS), which is the reason why some additional scales, such as the Profile of Community Psychiatry Clients, were developed. There are also some tools, such as the HoNOS, that indirectly measure needs for care, so they can be used as either a clinical or needs assessment tool.

Needs assessment tools are generally used by community psychiatry organizations and are also used to support changes to the organizations of countries’ related systems. The tools have already been used in order to assess the needs within clinical procedures, as well as at higher organizational levels in order to supplement services and direct programming (Royse & Drude, 1982 ). Different tools have good potential to evaluate community mental health services through assessing if patients’ needs have been met. Therefore, this study also aims at answering the question of which tool(s) can be most appropriate regarding different goals.

Within this review, we identified three systematic approaches to needs assessment which encompass different tools. The first is the DISC (Developing Individual Services in the Community) Framework (Smith, 1998 ), which includes the CAN and the Avon Self-Assessment Measure. The second is the Cumulative Needs for Care Monitor (Drukker et al., 2010 ), developed in order to choose the best treatment for each person. This one also uses the CAN and other more clinical tools and outcome measures (such as quality of life). The third is the Colorado Client Assessment Record (Ellis et al., 1984 ), which includes different measures of social functioning, such as the Denver Community of Mental Health Questionnaire, the Community Adjustment Profile, the Fort Logan Evaluation Screen, the Personal Role Skills Scale and the Global Assessment Scale.

This study has several strengths. First, we searched for as many tools and articles as possible. Second, we followed the standard rules of systematic and scoping reviews to present the data in a structured and non-biased manner: we thus searched for information extensively; the search was transparent and reproducible; the data were presented in a structured way. Finally, the scoping review was carried out, since the goal was not to compare and assess the quality of the evidence in the studies, but rather to review of all potential tools that can be used within the process of assessing the needs. Third, this study considered different populations, from severe mental disorders to other mental health problems, including addiction, which produced a strong overview of different tools and versions of the same tool used in other contexts. Fourth, the use of such tools also has a different basis depending on the goals of the system, so it can reflect the organization of care for mental health in a given country. The fifth strength of this work is that in addition to the original 242 articles within the review, we have also included all original sources for development of each of the 29 recognized tools.

This study also has some limitations. First, as the keywords are not same for every study, some studies could have been left out and therefore some tools might have been unrecognized. Second, our needs assessment review focuses on all people with mental health problems, even though the group of those with severe mental illness differs from the group with less severe mental health disorders. Therefore, no conclusion can be made on which tool is better for use in different population groups or disease severities. Third, we only included tools that assess the needs of people with mental health problems, although other tools for the general population could also potentially be useful. Fourth, some tools were developed and validated in only one country, so transferability is questionable or requires additional validation.

Since this scoping review provides insight into the evidence about the existence of different tools for needs assessment, it would also be valuable to conduct additional research on the level of each tool to see if it has already been validated and culturally adapted. To the best of our knowledge, the CAN is the most frequently used tool, and has been translated and adapted into more than 33 different languages (Phelan et al., 1995 ). Some of the tools reviewed in this study use items similar to the CAN, such as the Needs Assessment Scale (de Weert-van Oene et al., 2009 ). Some tools use the same items with a few additional ones, such as the Montreal Assessment of Needs Questionnaire (Tremblay et al., 2014 ), which shows even greater use of the CAN. Thus, the concepts in this latter tool are widely applied.

There are different fields in which certain needs must be addressed to deal with the mental health of the general population or the needs of the population with mental health problems, with the latter being our main focus. This review aimed to develop a tool for needs assessment that can be applied clinically and for research purposes. It is also vital to see what kind of tools can be used to assess needs for the purpose of a formative evaluation process, and the possibility of service development following the identification of actual needs (Makivić et al., 2021 ). Therefore, this article is valuable for a variety of final users, as it can be used by service providers at the level of health or social care, researchers, policymakers and other relevant stakeholders.

Moreover, it is also necessary to assess needs in the field of communication, especially targeting anti-stigma and anti-discrimination campaigns, and to assess the needs of educational systems (Kragelj et al., 2022 ) for the representation of mental health topics (Makivić et al., 2022 ). The use of different tools for assessing needs not only gives us the possibility of identifying such needs, but also establishes the possibility of meeting those needs when these tools are used within bio-psycho-socially oriented primary care or interdisciplinary-oriented mental health care. The assessment of needs at the individual level is important for the effective development of person-centred care plans (Martin et al., 2009 ). Patient-centred psychiatric practice is also needed to increase patient empowerment, which can be done with the help of a needs assessment process.

The review of all the tools for assessing different needs for people with mental health problems presented in this work is new, and therefore fills an important gap in the scientific knowledge of the needs assessment process in the field of mental health.

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Aoun, S., Pennebaker, D., & Wood, C. (2004). Assessing population need for mental health care: A review of approaches and predictors. Mental Health Services Research, 6 (1), 33–46. https://doi.org/10.1023/b:mhsr.0000011255.10887.59

Article   PubMed   Google Scholar  

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19–32. https://doi.org/10.1080/1364557032000119616

Article   Google Scholar  

Asadi-Lari, M., & Gray, D. (2005). Health needs assessment tools: Progress and potential. International Journal of Technology Assessment in Health Care, 21 (3), 288–297. https://doi.org/10.1017/s0266462305050385

Ashaye, O. A., Livingston, G., & Orrell, M. W. (2003). Does standardized needs assessment improve the outcome of psychiatric day hospital care for older people? A randomized controlled trial. Aging & Mental Health, 7 (3), 195–199. https://doi.org/10.1080/1360786031000101166

Article   CAS   Google Scholar  

Balacki, M. F. (1988). Assessing mental health needs in the rural community: A critique of assessment approaches. Issues in Mental Health Nursing, 9 (3), 299–315. https://doi.org/10.3109/01612848809140931

Article   PubMed   CAS   Google Scholar  

Bebbington, P., Brewin, C. R., Marsden, L., & Lesage, A. (1996). Measuring the need for psychiatric treatment in the general population: the community version of the MRC Needs for Care Assessment. Psychological Medicine, 26 (2), 229–236. https://doi.org/10.1017/s0033291700034620

Carter, M. F., Crosby, C., Geerthuis, S., & Startup, M. (1995). A client-centered assessment of need needs assessment. Journal of Mental Health, 4 (4), 383–394. https://doi.org/10.1080/09638239550037433

Carvacho, R., Carrasco, M., Lorca, M. B. F., & Miranda-Castillo, C. (2021). Met and unmet needs of dependent older people according to the Camberwell Assessment of Need for the Elderly (CANE): A scoping review. Revista Espanola De Geriatria y Gerontologia, 56 (4), 225–235. https://doi.org/10.1016/j.regg.2021.02.004

Davies, E. L., Gordon, A. L., Pelentsov, L. J., Hooper, K. J., & Esterman, A. J. (2018). Needs of individuals recovering from a first episode of mental illness: A scoping review. International Journal of Mental Health Nursing, 27 (5), 1326–1343. https://doi.org/10.1111/inm.12518

Davies, E. L., Gordon, A. L., Pelentsov, L. J., & Esterman, A. J. (2019). The supportive care needs of individuals recovering from first episode psychosis: A scoping review. Perspectives in Psychiatric Care, 55 (1), 6–14. https://doi.org/10.1111/ppc.12259

de Weert-van Oene, G. H., Havenaar, J. M., & Schrijvers, A. J. (2009). Self-assessment of need for help in patients undergoing psychiatric treatment. Psychiatry Research, 167 (3), 221–230. https://doi.org/10.1016/j.psychres.2008.02.001

Dobrzyńska, E., Rymaszewska, J., & Kiejna, A. (2008a). CANSAS-Krótka Ocena Potrzeb Camberwell oraz inne narzedzia oceny potrzeb osób z zaburzeniami psychicznymi [CANSAS-Camberwell Assessment of Need Short Appraisal Schedule and other needs of persons with mental disorders assessment tools]. Psychiatria Polska, 42 (4), 525–532.

PubMed   Google Scholar  

Dobrzyńska, E., Rymaszewska, J., & Kiejna, A. (2008b). Potrzeby osób z zaburzeniami psychicznymi–definicje, przeglad badań [Needs of persons with mental disorders-definitions and literature review]. Psychiatria Polska, 42 (4), 515–524.

Drukker, M., Bak, M., Campo, J.à, Driessen, G., Van Os, J., & Delespaul, P. (2010). The cumulative needs for care monitor: a unique monitoring system in the south of the Netherlands. Social Psychiatry and Psychiatric Epidemiology , 45 (4), 475–485. https://doi.org/10.1007/s00127-009-0088-3

Ellis, R. H., Wilson, N. Z., & Foster, F. M. (1984). Statewide treatment outcome assessment in Colorado: The Colorado Client Assessment Record (CCAR). Community Mental Health Journal, 20 (1), 72–89. https://doi.org/10.1007/BF00754105

Endacott, R. (1997). Clarifying the concept of need: A comparison of two approaches to concept analysis. Journal of Advanced Nursing, 25 (3), 471–476. https://doi.org/10.1046/j.1365-2648.1997.1997025471.x

Hamid, W. A., Lewis-Cole, K., Holloway, F., & Silverman, M. (2009). Older people with enduring mental illness: A needs assessment tool. Psychiatric Bulletin, 33 (3), 91–95. https://doi.org/10.1192/pb.bp.107.017392

Joska, J., & Flisher, A. J. (2005). The assessment of need for mental health services. Social Psychiatry and Psychiatric Epidemiology, 40 (7), 529–539. https://doi.org/10.1007/s00127-005-0920-3

Keulen-de Vos, M., & Schepers, K. (2016). Needs assessment in forensic patients: A review of instrument suites. International Journal of Forensic Mental Health, 15 (3), 283–300. https://doi.org/10.1080/14999013.2016.1152614

Kilbourne, A. M., Beck, K., Spaeth-Rublee, B., Ramanuj, P., O’Brien, R. W., Tomoyasu, N., & Pincus, H. A. (2018). Measuring and improving the quality of mental health care: A global perspective. World Psychiatry: Official Journal of the World Psychiatric Association (WPA), 17 (1), 30–38. https://doi.org/10.1002/wps.20482

Kragelj, A., Pahor, M., Zaletel-Kragelj, L., & Makivic, I. (2022). Learning needs assessment among professional workers in community mental health centres in Slovenia: Study protocol. South Eastern European Journal of Public Health , Special Volume No. 1 . https://doi.org/10.11576/seejph-5478

Larson, A., Frkovic, I., van Kooten-Prasad, M., & Manderson, L. (2001). Mental health needs assessment in Australia’s Culturally Diverse Society. Transcultural Psychiatry, 38 (3), 333–347. https://doi.org/10.1177/136346150103800304

Lasalvia, A., Stefani, B., & Ruggeri, M. (2000a). I bisogni di cura nei pazienti psichiatrici: una revisione sistematica della letteratura. I. Concetti generali e strumenti di valutazione. I bisogni di servizi [Therapeutic needs in psychiatric patients: a systematic review of the literature. I. General concepts and assessment measures. Needs for services]. Epidemiologia e Psichiatria Sociale , 9 (3), 190–213. https://doi.org/10.1017/s1121189x00007879

Lasalvia, A., Stefani, B., & Ruggeri, M. (2000b). I bisogni di cura nei pazienti psichiatrici: una revisione sistematica della letteratura. II. I bisogni di cura individuali [Needs for care in psychiatric patients: a systematic review. II. Needs for care on individual]. Epidemiologia e Psichiatria Sociale , 9 (4), 282–307. https://doi.org/10.1017/s1121189x00008411

Lasalvia, A., Ruggeri, M., Mazzi, M. A., & Dall'Agnola, R. B. (2000). The perception of needs for care in staff and patients in community-based mental health services. The South-Verona Outcome Project 3. Acta Psychiatrica Scandinavica , 102 (5), 366–375. https://doi.org/10.1034/j.1600-0447.2000.102005366.x

Le Grand, D., Kessler, E., & Reeves, B. (1996). The avon mental health measure. Mental Health Review Journal, 1 (4), 31–32. https://doi.org/10.1108/13619322199600042

Lesage, A. (2017). Populational and individual perspective on needs. Epidemiology and Psychiatric Sciences, 26 (6), 609–611. https://doi.org/10.1017/S204579601700049X

Article   PubMed   PubMed Central   CAS   Google Scholar  

Macpherson, R., Varah, M., Summerfield, L., Foy, C., & Slade, M. (2003). Staff and patient assessments of need in an epidemiologically representative sample of patients with psychosis-staff and patient assessments of need. Social Psychiatry and Psychiatric Epidemiology, 38 (11), 662–667. https://doi.org/10.1007/s00127-003-0669-5

Makivić, I., & Klemenc-Ketiš, Z. (2022). Development and validation of the scale for measuring biopsychosocial approach of family physicians to their patients. Family Medicine and Community Health, 10 (2), e001407. https://doi.org/10.1136/fmch-2021-001407

Article   PubMed   PubMed Central   Google Scholar  

Makivić, I., & Kragelj, A. (2023). Needs assessment tools at the field of mental health: Countries presentation. V: Conference abstract book: 11th European Conference on Mental Health [S. l. ]: Evipro, Str. Ljubljana, Slovenia, p. 184.

Makivić, I., Švab, V., & Selak, Š. (2021). Mental health needs assessment during the COVID-19 Pandemic: Consensus based on Delphi Study. Frontiers in Public Health, 9 , 732539. https://doi.org/10.3389/fpubh.2021.732539

Makivić, I., Kragelj, A., & Selak, Š. (2022). Educational needs assessment in the field of mental health: Review of the tools. V: 10th European conference on mental health: Conference abstract boo k. Evipro, Str. Lisbon, Portugal, Helsinki, pp. 183–184.

Marshall, M., Hogg, L. I., Gath, D. H., & Lockwood, A. (1995). The cardinal needs schedule-a modified version of the MRC needs for care assessment schedule. Psychological Medicine, 25 (3), 605–617. https://doi.org/10.1017/s0033291700033511

Martin, L., Hirdes, J. P., Morris, J. N., Montague, P., Rabinowitz, T., & Fries, B. E. (2009). Validating the Mental Health Assessment Protocols (MHAPs) in the Resident Assessment Instrument Mental Health (RAI-MH). Journal of Psychiatric and Mental Health Nursing, 16 (7), 646–653. https://doi.org/10.1111/j.1365-2850.2009.01429.x

McGorry, P. D., & Hamilton, M. P. (2016). Stepwise expansion of evidence-based care is needed for mental health reform. The Medical Journal of Australia, 204 (9), 351–353. https://doi.org/10.5694/mja16.00120

Meadows, G., Harvey, C., Fossey, E., & Burgess, P. (2000). Assessing perceived need for mental health care in a community survey: Development of the Perceived Need for Care Questionnaire (PNCQ). Social Psychiatry and Psychiatric Epidemiology, 35 (9), 427–435. https://doi.org/10.1007/s001270050260

Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18 (1), 143. https://doi.org/10.1186/s12874-018-0611-x

Phelan, M., Slade, M., Thornicroft, G., Dunn, G., Holloway, F., Wykes, T., Strathdee, G., Loftus, L., McCrone, P., & Hayward, P. (1995). The Camberwell assessment of need: The validity and reliability of an instrument to assess the needs of people with severe mental illness. The British Journal of Psychiatry: The Journal of Mental Science, 167 (5), 589–595. https://doi.org/10.1192/bjp.167.5.589

Ravaghi, H., Guisset, A. L., Elfeky, S., Nasir, N., Khani, S., Ahmadnezhad, E., & Abdi, Z. (2023). A scoping review of community health needs and assets assessment: Concepts, rationale, tools and uses. BMC Health Services Research, 23 (1), 44. https://doi.org/10.1186/s12913-022-08983-3

Royse, D., & Drude, K. (1982). Mental health needs assessment: Beware of false promises. Community Mental Health Journal, 18 (2), 97–106. https://doi.org/10.1007/BF00754454

Slade M. (1994). Needs assessment. Involvement of staff and users will help to meet needs. The British journal of psychiatry: the journal of mental science, 165 (3), 293–296. https://doi.org/10.1192/bjp.165.3.293

Smith, H. (1998). Needs assessment in mental health services: The DISC framework. Journal of Public Health Medicine, 20 (2), 154–160. https://doi.org/10.1093/oxfordjournals.pubmed.a024736

Stewart, R. (1979). The nature of needs assessment in community mental health. Community Mental Health Journal, 15 , 287–295. https://doi.org/10.1007/BF00778708

Thornicroft, G. (1991). The concept of case management for long-term mental illness. International Review of Psychiatry, 3 (1), 125–132. https://doi.org/10.3109/0954026910906752

Thornicroft, G., & Slade, M. (2002). Comparing needs assessed by staff and by service users: Paternalism or partnership in mental health? Epidemiologia e Psichiatria Sociale, 11 (3), 186–191. https://doi.org/10.1017/s1121189x00005704

Tremblay, J., Bamvita, J. M., Grenier, G., & Fleury, M. J. (2014). Utility of the Montreal assessment of need questionnaire for community mental health planning. The Journal of Nervous and Mental Disease, 202 (9), 677–687. https://doi.org/10.1097/NMD.0000000000000180

Witkin, B. R., & Altschuld, J. W. (1995). Planning and Conducting Needs Assessments: A Practical Guide . Sage Publications. https://doi.org/10.1177/002087289603900316

Wykes, T., Bell, A., Carr, S., Coldham, T., Gilbody, S., Hotopf, M., Johnson, S., Kabir, T., Pinfold, V., Sweeney, A., Jones, P. B., & Creswell, C. (2021). Shared goals for mental health research: what, why and when for the 2020s. Journal of Mental Health (Abingdon, England) , 1–9. Advance online publication. https://doi.org/10.1080/09638237.2021.1898552

Yu, J. Y., Dong, Z. Q., Liu, Y., Liu, Z. H., Chen, L., Wang, J., Huang, M. J., Mo, L. L., Luo, S. X., Wang, Y., Guo, W. J., He, N., Chen, R., & Zhang, L. (2019). Patient-doctor concordance of perceived mental health service needs in Chinese hospitalized patients: a cross-sectional study. Annals of Palliative Medicine , 8 (4), 442–450. https://doi.org/10.21037/apm.2019.08.06

Download references

This work was supported by the Slovenian Research Agency (project No.: Z3-2652).[Display Image Removed]. Project was funded by the Reseach agency. Only IM was paid from this. But the project didn't finance the outcomes and the goal of publishing this work was solely for the research purposes.

Author information

Authors and affiliations.

National Institute of Public Health, Ljubljana, Slovenia

Irena Makivić & Anja Kragelj

Section of Psychiatry, Department of Neuroscience, Biomedicine and Movement Sciences, University of Verona, Verona, Italy

Antonio Lasalvia

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: [Irena Makivić]; Methodology: [Irena Makivić]; Formal analysis and investigation: [Irena Makivić], [Anja Kragelj]; Writing—original draft preparation: [Irena Makivić]; Writing—review and editing: [Irena Makivić], [Anja Kragelj], [Antonio Lasalvia]; Funding acquisition: [Irena Makivić]; Resources: [Irena Makivić]; Supervision: [Antonio Lasalvia].

Corresponding author

Correspondence to Irena Makivić .

Ethics declarations

There are no financial or non-financial interests that are directly or indirectly related to the work submitted for publication. All authors have no relevant financial or non-financial interests to disclose.

There was no need to apply for ethical committee or for informed consent, since there were no humans directly involved in this scoping review.

Embargoes on data sharing are permitted.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 432 KB)

Supplementary file2 (pdf 556 kb), rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Makivić, I., Kragelj, A. & Lasalvia, A. Quantitative needs assessment tools for people with mental health problems: a systematic scoping review. Curr Psychol (2024). https://doi.org/10.1007/s12144-024-05817-9

Download citation

Accepted : 27 February 2024

Published : 12 March 2024

DOI : https://doi.org/10.1007/s12144-024-05817-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Needs assessment
• Mental health
• Patient-centred care
• Find a journal
• Publish with us
• Track your research

Change Password

Your password must have 6 characters or more:.

• a lower case character, 
• an upper case character, 
• a special character 

Password Changed Successfully

Your password has been changed

Create your account

Forget yout password.

Enter your email address below and we will send you the reset instructions

If the address matches an existing account you will receive an email with instructions to reset your password

Forgot your Username?

Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

• March 01, 2024 | VOL. 75, NO. 3 CURRENT ISSUE pp.203-304
• February 01, 2024 | VOL. 75, NO. 2 pp.107-201
• January 01, 2024 | VOL. 75, NO. 1 pp.1-71

The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use , including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

Mixed-Methods Designs in Mental Health Services Research: A Review

• Lawrence A. Palinkas , Ph.D. ,
• Sarah M. Horwitz , Ph.D. ,
• Patricia Chamberlain , Ph.D. ,
• Michael S. Hurlburt , Ph.D. , and
• John Landsverk , Ph.D.

Search for more papers by this author

Despite increased calls for use of mixed-methods designs in mental health services research, how and why such methods are being used and whether there are any consistent patterns that might indicate a consensus about how such methods can and should be used are unclear.

Use of mixed methods was examined in 50 peer-reviewed journal articles found by searching PubMed Central and 60 National Institutes of Health (NIH)-funded projects found by searching the CRISP database over five years (2005–2009). Studies were coded for aims and the rationale, structure, function, and process for using mixed methods.

A notable increase was observed in articles published and grants funded over the study period. However, most did not provide an explicit rationale for using mixed methods, and 74% gave priority to use of quantitative methods. Mixed methods were used to accomplish five distinct types of study aims (assess needs for services, examine existing services, develop new or adapt existing services, evaluate services in randomized controlled trials, and examine service implementation), with three categories of rationale, seven structural arrangements based on timing and weighting of methods, five functions of mixed methods, and three ways of linking quantitative and qualitative data. Each study aim was associated with a specific pattern of use of mixed methods, and four common patterns were identified.

Conclusions:

These studies offer guidance for continued progress in integrating qualitative and quantitative methods in mental health services research consistent with efforts by NIH and other funding agencies to promote their use. ( Psychiatric Services 62:255–263, 2011)

In the past decade, mental health services researchers have increasingly used qualitative methods in combination with quantitative methods ( 1 , 2 ). This use of mixed methods has been partly driven by theoretical models that encourage assessment of consumer perspectives and of contextual influences on disparities in the delivery of mental health services and the dissemination and implementation of evidence-based practices ( 3 , 4 ). These models call for research designs that use quantitative and qualitative data collection and analysis for a better understanding of a research problem than might be possible with use of either methodological approach alone ( 5 , 6 ). Numerous typologies and guidelines for the use of mixed-methods designs exist in the fields of nursing ( 7 , 8 ), evaluation ( 9 , 10 ), public health ( 11 , 12 ), primary care ( 13 ), education ( 14 ), and the social and behavioral sciences ( 5 , 15 ).

As Robins and colleagues ( 1 ) have observed, however, there has been little guidance in mental health services research on how to blend quantitative and qualitative methods to build upon the strengths of their respective epistemologies. Such guidance has been limited by the lack of consensus on the criteria that might be used to evaluate the quality of such research ( 5 ). From a policy perspective, the impact of the efforts of the National Institute of Mental Health (NIMH) ( 3 , 4 ) and other institutes of the National Institutes of Health (NIH) and funding agencies in encouraging the use of mixed methods in mental health services research also remains poorly understood.

To address these issues, we examined the application of mixed-methods designs in a sample of mental health services research studies published in peer-reviewed journals and in NIMH-funded research projects over five years. Our aim was to determine how and why such methods were being used and whether there are any consistent patterns that might indicate a consensus among researchers as to how such methods can and should be used. This aim is viewed as an initial step toward the development of standards for effective uses of mixed methods in mental health services research and articulation of criteria for evaluating the quality and impact of this research.

We conducted a literature review of mental health services research publications over a five-year period (January 2005 to September 2009), using the PubMed Central database and the following search terms: mental health services, mixed methods, and qualitative methods. Data were taken from the full text of each research article. Articles identified as potential candidates for inclusion had to report empirical research and meet one of the following selection criteria: a study specifically identified as a mixed-methods study in the title or abstract or through keywords; a qualitative study conducted as part of a larger project, including a randomized controlled trial, that also included use of quantitative methods; or a study that “quantitized” qualitative data ( 16 ) or “qualitized” quantitative data ( 17 ). On the basis of criteria used by McKibbon and Gadd ( 18 ) and Cresswell and Plano Clark ( 5 ), the analysis had to be fairly substantial; for example, a simple descriptive analysis of baseline demographic characteristics of participants was not sufficient to be included as a mixed-methods study. Further, qualitative studies that were not clearly linked to quantitative studies or methods were excluded from our review.

Using the same criteria and search terms, we also reviewed the NIH CRISP database (Computer Retrieval of Information on Scientific Projects) of projects funded over the same five-year period. Projects were limited to R series (independent research awards), F series (predissertation research awards), and K series (career development awards) grants. Data were taken from only the project descriptions provided by the applicant and contained in the database.

Using typologies employed in other fields of inquiry ( 5 – 7 , 9 ), we next assessed the use of mixed methods in each study to determine the study aims, rationale, structure, function, and process. Study aims referred to the objectives of the overall project that included both quantitative and qualitative studies or methods. The rationale for using mixed methods included conceptual reasons, such as exploration and confirmation ( 5 ), breadth and depth of understanding ( 19 ), and inductive and deductive theoretical drive ( 20 ). Pragmatic reasons for using mixed methods, such as addressing the weaknesses of one method by use of the other, and suitability to address research questions were also examined. Assessment of the structure of the research design was based on Morse's ( 7 ) taxonomy, which gives emphasis to timing (for example, using methods in sequence [represented by a → symbol] versus using them simultaneously [represented by a + symbol]) and to weighting (for example, primary method [represented by capital letters such as QUAN] versus secondary method [represented in lowercase letters such as qual]).

Assessment of the function of mixed methods was based on whether the two methods were being used to answer the same question or to answer related questions and whether they were used to achieve convergence, complementarity, expansion, development, or sampling ( 9 ). Finally, the process or strategies for combining qualitative and quantitative methods were assessed with the typology proposed by Cresswell and Plano Clark ( 5 ): merging or converging the two methods by actually bringing them together in the analysis or interpretation phase, connecting the two methods by having one build upon the results obtained by the other, or embedding one data set within the other so that one type of method provides a supportive role for the other method.

Our search identified 50 articles and 67 NIH-funded research projects published or funded between 2005 and 2009 that met our criteria for analysis. Seven of the NIH projects were excluded from further review because of missing data on the use of mixed methods. Three of the publications were based on one of the NIH-funded projects, and two other publications were based on one funded project each. Any redundant aims or strategies for combining qualitative and quantitative methods identified in linked publications and projects were counted only once in our analysis.

A list of the 26 journals in which the articles were published and the journals' impact factors (IFs) is presented in Table 1 . One-fifth of the articles were published in Psychiatric Services . The 2008 IFs of the journals for which information was available ranged from .74 ( Psychiatric Rehabilitation Journal ) to 4.84 ( Journal of the American Academy of Child and Adolescent Psychiatry ). Twenty-one of the 50 articles (42%) had an IF of 2.0 or greater. Of the funded grants, three were predissertation research grants (F31s), 28 were career development awards (K01, K08, K23, K24, and K99), and 29 were independent research awards (R01, R03, R18, R21, R24, and R34).

Table 2 presents the year of publication for the 50 articles and the start date of the 60 funded projects. Sixteen of the projects funded during this period had a start date before 2005. The smaller numbers of publications and of projects in 2009 reflect the shorter period of observation (nine months) for that year. There was an exponential increase in the number of publications between 2005 and 2008, and the number of grants from 2005 to 2009 was more than twice that of the previous five-year period (2000–2004).

Table 3 summarizes for comparison the use of mixed-methods designs on the basis of study aims. Our analyses revealed the use of mixed methods to accomplish five distinct types of study aims and three categories of rationale. We further identified seven structural arrangements, five uses or functions of mixed methods, and three ways of linking quantitative and qualitative data together. Some papers and projects included more than one objective, structure, or function; hence the raw numbers may occasionally sum to more than the total number of studies examined. Twelve of the 50 articles presented qualitative data only but were part of larger studies that included the use of quantitative measures. Further, we identified four commonly used designs, with each design associated with a specific aim or set of aims ( Figure 1 ).

As shown in Table 3 , the largest number of publications and projects (41 of 110, 37%) used mixed methods in observational or quasi-experimental studies of existing services. Almost one-quarter (24%) used mixed methods to study the implementation and dissemination of evidence-based practices. Mixed methods were also used to develop evidence-based practices, treatment, and interventions (17%); to conduct randomized controlled trials of interventions (14%); or to assess the needs of populations for mental health services (14%). Six studies had more than one aim (for example, two studies conducted a needs assessment before developing new interventions, and two studies examined implementation of an evidence-based practice within the context of a randomized controlled trial examining the practice's effectiveness.

Mixed-methods rationale

Forty-one of the 60 project abstracts (68%) and 25 of the 50 published articles (50%) did not provide an explicit rationale for the use of mixed methods; consequently, the rationale was inferred from statements found in project objectives. Of the 25 published articles that did provide an explicit rationale, only 11 provided one or more citations to justify use of mixed methods. The most common reason (93% of all articles and projects) for using mixed methods was based on the specific objectives of the study (for example, qualitative methods were needed for exploration or depth of understanding or quantitative methods were needed to test hypotheses). In other instances, use of mixed methods was dictated by the nature of the data; studies that included a focus on variables related to values and beliefs, the process of service delivery, or the context in which services are delivered relied on qualitative methods to describe and examine these phenomena. In 9% of articles and projects, investigators specifically indicated that both methods were used so that the strengths of one method could offset the weaknesses of the other ( Table 3 ).

Mixed-methods structure

The majority (58%) of the publications and projects used the methods in sequence, with qualitative methods more often preceding quantitative methods. Quantitative methods were the primary or dominant method in 74% of the publications and projects reviewed, and in 16 studies, qualitative and quantitative methods were given equal weight. In seven of the published studies, qualitative analyses were conducted on one or two open-ended questions attached to a survey, and 17 of the 50 published studies (34%) provided no references justifying their procedures for qualitative data collection or analysis. Only one published study ( 21 ) provided a figure that illustrated the timing and weighting of qualitative and quantitative data collection and analysis, and none used terms like QUAN and qual to describe this structure.

In studies that aimed to assess needs for mental health services, examine existing services, or develop new services or adapt existing services to new populations, sequential designs were used two to four times more frequently than simultaneous designs. The latter type of design was more commonly used in randomized controlled trials and in implementation studies.

Mixed-methods functions

Our review of the publications and projects revealed five distinct functions of mixing methods ( Table 3 ). The first function was convergence, in which qualitative and quantitative methods were used sequentially or simultaneously to answer the same question, either through triangulation (that is, the simultaneous use of one type of data to validate or confirm conclusions reached from analysis of the other type of data) or transformation (that is, the sequential quantification of qualitative data or use of qualitative techniques to transform quantitative data). For instance, Griswold and colleagues ( 22 ) triangulated quantitative trends in functional and health outcomes of psychiatric emergency department patients with qualitative findings of perceived benefits of care management and the value of integrated medical and mental health care to determine whether both types of data provided support for the effectiveness of a care management intervention (QUAN + QUAL). Using the technique of concept mapping ( 23 ), Aarons and colleagues ( 24 ) collected qualitative data on factors likely to have an impact on implementation of evidence-based practices in public-sector mental health settings. These data were then entered in a software program that uses multidimensional scaling and hierarchical cluster analysis to generate a visual display of statement clusters (QUAL → quan).

A second function of integrating quantitative and qualitative methods was complementarity, in which each method was used to answer related questions for the purpose of evaluation or elaboration. This function was evident in a majority (65%) of the published studies and projects examined. In evaluative designs, quantitative data were used to evaluate outcomes, whereas qualitative data were used to evaluate process. For instance, Bearsley-Smith and colleagues ( 25 ) described the use of quantitative methods to investigate the impact on clinical care of implementing interpersonal psychotherapy for adolescents within a rural mental health service and the use of qualitative methods to record the process and challenges (that is, feasibility, acceptability, and sustainability) associated with implementation and evaluation (QUAN + qual). In elaborative designs, qualitative methods were used to provide depth of understanding and quantitative methods were used to provide breadth of understanding. For instance, in a longitudinal study of mental health consumer-run organizations, Janzen and colleagues ( 26 ) used a quantitative tracking log for breadth of information about system-level activities and outcomes and key informant interviews and focus groups for greater insight into the impacts of these activities (QUAL + quan).

A third function of integrating qualitative and quantitative methods was expansion, in which one method was used in sequence to answer questions raised by the other method. This function was evident in 24% of the published studies and projects examined. In each instance, qualitative data were used to explain findings from the analyses of quantitative data. Brunette and colleagues ( 27 ) interviewed key informants and conducted ethnographic observations of implementation efforts to understand why some agencies adhered to established principles for integrated dual disorders treatment and others did not (QUAN + qual).

A fourth function of mixed methods was development, in which qualitative methods were used sequentially to identify form and content of items to be used in a quantitative study (for example, survey questions), to create a conceptual framework for generating hypotheses to be tested by using quantitative methods, or to develop new interventions or adapt existing interventions to new populations (qual → QUAN). This function was used in 34% of the published studies and projects. Blasinsky and colleagues ( 28 ) used qualitative findings from site visits to develop quantitative rating scales to construct predictors of outcomes and sustainability of a collaborative care intervention for older adults who had major depressive disorder or dysthymia. Green and colleagues ( 29 ) used qualitative data to generate a theoretical model of how relationships with clinics and clinicians' approach affect quality of life and recovery from serious mental illness and then tested the model using questionnaire data and health-plan and interview-based data in a covariance structure model. Several of the research projects funded through the R34 mechanism (for example, MH074509-01, Kilbourne, principal investigator [PI]; MH078583-01, Druss, PI; and MH073087-01, Lewis-Fernandez, PI) used qualitative data obtained from focus groups of consumers and providers to develop or adapt interventions for clients with specific conditions (for example, bipolar disorder, chronic medical conditions, and depressive disorders) (qual − QUAN).

The final function of mixed methods was sampling, the sequential use of one method to identify a sample of participants for research that uses the other method. This technique was used in only 7% of all studies. One form of sampling was the sequential use of quantitative data to identify potential participants for a qualitative study (quan − QUAL). For instance, Aarons and Palinkas ( 30 ) purposefully sampled candidates for qualitative interviews who had the most positive or most negative views of an evidence-based practice on the basis of a Web-based quantitative survey. The other form of sampling used qualitative data to identify samples of participants for quantitative analysis (qual − QUAN). Woltmann and colleagues ( 31 ) created categories of low, medium, and high staff turnover on the basis of staff perceptions of relevance of turnover obtained from qualitative interviews and then quantitatively examined the relationship between these turnover categories and implementation outcomes (qual + QUAN).

Only six of the published studies and none of the project abstracts explicitly referred to the function of mixed methods by using terms such as triangulation (four published studies) or complementarity (two published studies). As expected, the development function was used in a majority (84%) of studies that aimed to develop new practices or adapt existing practices to new populations. A majority of observational and quasi-experimental studies of existing services (71%), randomized controlled trials (67%), implementation studies (65%), and needs assessment studies (60%) utilized mixed methods for the purposes of answering related questions in complementary fashion. The use of one set of methods to explain the results of a study using another set of methods appears to have been limited to implementation studies (46%), randomized controlled trial evaluations (40%), and studies of existing services (20%).

Process of mixing methods

The final characteristic of mixed-methods designs that we examined was the process of mixing the quantitative and qualitative methods. The largest percentage (47%) of articles and projects sought to connect the data sets ( Table 3 ). This occurs when the analysis of one data set leads to (and thereby connects to) the need for the other data set, such as when quantitative results lead to the subsequent collection and analysis of qualitative data (that is, expansion) or when qualitative results are used to build to the subsequent collection and analysis of quantitative data, (for example, development) ( 5 ). For instance, Frueh and colleagues ( 32 ) conducted focus groups to obtain information on the target population, their providers, and state-funded mental health systems that would enable the researchers to further adapt and improve a cognitive-behavioral therapy-based intervention for treatment of posttraumatic stress disorder before implementing it (qual → QUAN). This type of mixing was found in almost all of the studies with aims to develop new practices or adapt existing practices to new populations; it was also more likely to be found in needs assessment and studies of existing services than in randomized controlled trials or implementation studies.

Over one-third (37%) of the studies merged the knowledge gained from the quantitative and qualitative data, either during the interpretation phase when two sets of results that had been analyzed separately were brought together or during the analysis phase when one type of data was transformed into the other type by consolidating the data into new variables ( 5 ). This type of mixing was found in slightly less than half of the needs assessment, observational, and implementation studies. For instance, Lucksted and colleagues ( 33 ) reported that a qualitative analysis of responses to an open-ended postintervention question supported the quantitative findings of the benefits of a relapse prevention and wellness program (QUAN + qual).

The embedding of small qualitative or qualitative-quantitative studies within larger quantitative studies was observed in 35% of the published studies and projects reviewed and described as “nested designs” in six of the studies. This type of mixing was more commonly found in randomized controlled trials and in implementation studies, where qualitative studies of treatment or implementation process or context were embedded within larger quantitative studies of treatment or implementation outcome. For instance, to better understand the essential components of the patient-provider relationship in a public health setting, Sajatovic and colleagues ( 34 ) conducted a qualitative investigation of patients' attitudes toward a collaborative care model and how individuals with bipolar disorder perceive treatment adherence within the context of a randomized controlled trial evaluating a collaborative practice model (QUAN + qual).

In 20% of published studies, more than one process was evident. For instance, Proctor and colleagues ( 35 ) connected the data by generating frequencies and rankings of qualitative data on perceptions of competing psychosocial problems collected from a community sample of 49 clients with a history of depression. These data were then merged with quantitative measures of depression status obtained through administration of the Patient Health Questionnaire-9 to explore the relationship of depression severity to problem categories and ranks.

The results of our analysis indicate that there has been substantial progress in using mixed-methods designs in mental health services research in response to efforts by NIMH ( 2 , 3 ) and other funding agencies to promote their use. Evidence for this progress is found in the increasing number of research projects that use mixed methods. The number of projects with mixed-methods designs funded over the five-year study period was more than twice the number that began in the previous five-year period (2000–2004). Furthermore, a majority (52%) of these funded projects were predissertation or career development awards used by junior and midlevel investigators to acquire expertise in mixed-methods research.

We also observed a notable increase in the number of studies based on mixed-methods designs published each year during this five-year period. The number of published mental health services research studies with mixed-methods designs increased by 67% between 2005 and 2006, by 80% between 2006 and 2007, and by 155% between 2007 and 2008. Furthermore, 21 of the 50 published studies (42%) that we reviewed appeared in journals with 2008 IFs of 2.0 or higher, including ten articles published in Psychiatric Services; four articles appeared in a journal with an IF of 4.0 or higher. In contrast, McKibbon and Gadd ( 18 ) reported that only 11 of 37 (30%) mixed-methods studies of health services appeared in a journal with an IF of 2.0 or higher in the year 2000.

Despite this progress, however, our review also suggests that there is room for improvement in use of mixed-methods designs. Most studies did not make explicit or provide support for the reasons for choosing a mixed-methods design; rather, we were forced to infer the rationale based on statements explaining what the methods were used for. Researchers may have felt that such explicit statements were as unnecessary as statements explaining the rationale for using certain quantitative methods, such as analysis of variance or survival analysis. However, the absence of an explicit rationale may also reflect a lack of understanding or appreciation of mixed-methods designs or a decision to use them without necessarily integrating or “mixing” them ( 5 , 6 ).

Most studies failed to provide explicit descriptions of the design structure or function that used terminology found in the mixed-methods literature; use of such terminology is consistent with the general standards for high-quality mixed-methods research recommended by Cresswell and Plano Clark ( 5 ). Further, three-fourths of the 50 published studies reviewed assigned priority to the use of quantitative methods, seven of the studies performed qualitative analyses of one or two open-ended questions attached to a survey, and 17 of the studies provided no references justifying their procedures for qualitative data collection or analysis. This may reflect an underappreciation of qualitative methods, as Robins and colleagues ( 1 ) have argued, or it may reflect a greater need for quantitative methods at the present time.

Although it was beyond the scope of this review to determine whether each study used mixed methods in effective ways, we note that each study was subjected to rigorous peer review before being published or funded, and each was judged by this process to make a valuable contribution to the field of mental health services research. These studies also provide evidence of meaningful and sensible variations in mixed-methods approaches to achieving various kinds of study aims and offer some guidance for integrating quantitative and qualitative methods in mental health services research. For instance, the choice of a mixed-methods design appears to be dictated by the nature of the questions being asked by mental health services researchers. Qualitative methods were used to explore a phenomenon when there was little or no previous research or to examine that phenomenon in depth, whereas quantitative methods were used to confirm hypotheses or examine the generalizability of the phenomenon and its associated predictors.

A majority of studies aiming to develop new practices or adapt existing practices to new populations had the same structure (beginning with a small qualitative study before developing or adapting the practice that was to be evaluated by using quantitative methods, which was found in 84% of the studies and projects) and the same process (connecting the findings of one set of methods with those of another set, which was found in 90% of the studies and projects). These studies reflect a growing awareness of the need to incorporate the preferences and perspectives of both service consumers and providers to ensure that new practices will be acceptable as well as feasible ( 32 , 36 – 39 ).

Studies of existing services also tended to be sequential in structure, with qualitative methods used to elaborate or explain the findings of quantitative studies. In the majority of these studies, the process of mixing methods involved either merging two sets of data to achieve convergence or connecting them to achieve expansion ( 5 ). A similar pattern was observed in studies that aimed to explore issues related to the needs for mental health services or provide more depth to our understanding of those needs. Such studies also appeared more likely to transform or “quantitize” qualitative data ( 24 , 35 ).

Randomized controlled trials and studies of implementation also shared similar patterns in use of mixed methods, including simultaneous use of both methods to achieve complementarity by embedding a qualitative or qualitative-quantitative study within a larger quantitative study, such as a randomized controlled trial. In the randomized controlled trials, qualitative methods were usually used to evaluate the process of providing the practice or intervention, whereas quantitative methods were used to evaluate the outcomes ( 25 , 40 ). In implementation research studies, qualitative methods were used to explore or provide depth to understanding barriers and facilitators of intervention implementation, whereas quantitative methods were used to confirm hypotheses and provide breadth to understanding by assessing the generalizability of findings ( 41 , 42 ).

The choice of mixed-methods designs was also dictated by how the individual questions being addressed by each method were related to one another. Studies that used different types of data to answer the same question reflected the function of convergence in a simultaneous structure, where data were merged for the purpose of triangulation, or a sequential structure, where qualitative data were transformed into quantitative data. Studies that used different types of data to answer related questions reflected the function of complementarity, in which quantitative methods were used to measure outcomes, describe content (for example, fidelity of services used and the nature of the mental health problem), and provide breadth (generalizability) of understanding, whereas qualitative methods were used to evaluate the process of service delivery ( 43 – 45 ), describe context (for example, setting) ( 26 , 34 , 46 ), describe consumer values or attitudes ( 35 , 42 , 47 ), and provide depth (meaning) of understanding ( 28 , 48 ) in a simultaneous structure and embedded data process. Expansion, development, and sampling were also used to provide answers to related questions that could not be answered by one method alone, usually in a sequential structure in which data sets were merged or connected together ( 24 , 30 , 37 ).

Finally, the choice of design appears to be based on the strengths of one method relative to the weaknesses of the other. For instance, expansion was used to explain findings based on quantitative data with qualitative data because explanation was not possible with the quantitative methods alone ( 25 , 27 , 40 ). In convergence, both sets of methods were used to confirm or validate one another, especially in instances where limited samples precluded testing of hypotheses with sufficient statistical power ( 30 , 49 ) and where limitations to qualitative data collection raised concerns about objectivity and transferability of results. In studies developing new methods, conceptual models, and interventions, qualitative methods also served to enhance quantitative analysis by laying the groundwork essential for more valid measurement and theory and more effective, usable, and sustainable interventions ( 37 ). Sampling also worked to enhance validity by using qualitative methods to enhance quantitative methods by developing targeted comparisons or by using quantitative methods to enhance qualitative methods by establishing criteria for purposeful sampling ( 36 ).

In summary, the choice of a mixed-methods design appears to be associated with three considerations: the nature of the question being asked (inductive-exploratory or deductive-confirmatory), how the questions being addressed by each method are related to one another, and the strengths of each method relative to the weaknesses of the other.

Caution should be exercised in interpreting these findings given limitations in our study design and analysis. Despite our efforts to be comprehensive in the search process and to select studies and projects on the basis of criteria with face validity, we undoubtedly excluded several articles or projects that used mixed methods. For example, we may have excluded mixed-methods projects listed in the CRISP database that did not specify use of qualitative or mixed methods in the abstracts. We may have also excluded published articles with qualitative data that were part of larger, primarily quantitative studies if the articles did not reference the larger studies, or we may have excluded articles not listed in PubMed Central. In the absence of explicit information, we were often forced to infer the structure, rationale, and function of the design based on statements contained in the available material. Similarly, the CRISP abstracts describe only what the investigators proposed to do with mixed methods and do not indicate what was actually done. Our use of existing typologies of structure, function, and process were intended to serve as a starting point in our analysis rather than an attempt to “pigeon-hole” each study into a specific typology. Our assessment of the progress made in the application of mixed-methods designs in response to calls for their use by funding agencies did not include indicators of whether these efforts had produced more useful, incisive, or insightful knowledge for the purpose of addressing mental health services questions and problems. Such an assessment would require comparisons with the products of studies based on monomethod designs, which was beyond the scope of this study.

Finally, it should be noted that the typology of mixed-methods use does not represent a set of standards for using mixed methods per se but is an important first step toward the development of such standards. Typologies by themselves do not explain why a particular method should be used and how to use a method appropriately. However, as Teddlie and Tashakkori ( 6 ) observed, there are five reasons or benefits to developing such a typology: typologies help to provide the field with an organizational structure, they provide examples of research designs that are clearly distinct from either qualitative or quantitative research designs, they help to establish a common language for the field, they help researchers decide how to proceed when designing their studies, and they are useful as a pedagogical tool. A consensus conference or workshop bringing together experts in mixed methods and mental health services research to evaluate the empirically generated typology found in current patterns of mixed-methods use would appear to be the next logical step in developing a set of standards. Such standards would also be required to adhere to the epistemological foundations of each method when used separately (for example, whether appropriate considerations are made to ensure the generalizability of quantitative results or theoretical saturation of qualitative data and whether each method is appropriately matched to the inductive or deductive theoretical drive of the study) and when combined (for example, whether the knowledge gained when using the two methods together is more insightful and of greater value than the knowledge gained when using them separately).

Conclusions

Despite the limitations described above, the findings suggest an increasing use of mixed-methods designs to address changing priorities in mental health services research and a consensus as to how such methods should be applied. The lack of explicit statements explaining the rationale for using mixed methods and the evident priority assigned to quantitative methods suggest that there is room for improvement. However, these studies appear to utilize a common set of designs and provide guidance for using mixed methods, with varying approaches based on the nature of the question being asked (exploratory or confirmatory), how questions being addressed by each method are related to one another, and the strengths of each method relative to the weaknesses of the other.

Acknowledgments and disclosures

This study was funded through NIMH grant P50-MH50313-07.

The authors report no competing interests.

1 Robins CS , Ware NC , dosReis S , et al. : Dialogues on mixed methods and mental health services research: anticipating challenges, building solutions . Psychiatric Services 59:727–731, 2008 Link ,  Google Scholar

2 Hopper K : Qualitative and quantitative research: two cultures . Psychiatric Services 59:711, 2008 Link ,  Google Scholar

3 Hoagwood K , Jensen PS , Petti T , et al. : Outcomes of mental health care for children and adolescents: I. a comprehensive conceptual model . Journal of the American Academy of Child and Adolescent Psychiatry 35:1055–1063, 1996 Crossref , Medline ,  Google Scholar

4 Hohmann AA : A contextual model for clinical mental health effectiveness research . Mental Health Services Research 1:83–91, 1999 Crossref ,  Google Scholar

5 Cresswell JW , Plano Clark VL : Designing and Conducting Mixed Method Research . Thousand Oaks, Calif, Sage, 2007 Google Scholar

6 Teddlie C , Tashakkori A : Major issues and controversies in the use of mixed methods in the social and behavioral sciences; in Handbook of Mixed Methods in the Social and Behavioral Sciences . Edited by Tashakkori ATeddlie C Thousand Oaks, Calif, Sage, 2003 Google Scholar

7 Morse JM : Approaches to qualitative-quantitative methodological triangulation . Nursing Research 40:120–123, 1991 Crossref , Medline ,  Google Scholar

8 Sandelowski M : Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed method studies . Research in Nursing and Health 23:246–255, 2000 Crossref , Medline ,  Google Scholar

9 Greene JC , Caracelli VJ , Graham WF : Toward a conceptual framework for mixed method evaluation designs . Educational Evaluation and Policy Analysis 11:255–274, 1989 Crossref ,  Google Scholar

10 Patton MQ : Qualitative Evaluation and Research Methods , 3rd ed. Newbury Park, Calif, Sage, 2002 Google Scholar

11 Morgan DL : Practical strategies for combining qualitative and quantitative methods: applications to health research . Qualitative Health Research 8:262–276, 1998 Crossref ,  Google Scholar

12 Steckler A , McLeroy KR , Goodman RM , et al. : Toward integrating qualitative and quantitative methods: an introduction . Health Education Quarterly 19:1–8, 1992 Crossref , Medline ,  Google Scholar

13 Cresswell JW , Fetters MD , Ivankova NV : Designing a mixed methods study in primary care . Annals of Family Medicine 2:7–12, 2004 Crossref , Medline ,  Google Scholar

14 Cresswell JW : Mixed method research: introduction and application; in Handbook of Educational Policy . Edited by Cizek GJ San Diego, Academic Press, 1999 Crossref ,  Google Scholar

15 Waszak C , Sines MC : Mixed methods in psychological research; in Handbook of Mixed Methods in the Social and Behavioral Sciences . Edited by Tashakkori ATeddlie C Thousand Oaks, Calif, Sage 2003 Google Scholar

16 Miles MB , Huberman AM : Qualitative Data Analysis: An Expanded Sourcebook , 2nd ed. Thousand Oaks, Calif, Sage, 1994 Google Scholar

17 Tashakkori A , Teddlie C : Mixed Methodology: Combining the Qualitative and Quantitative Approaches . Thousand Oaks, Calif, Sage, 1998 Google Scholar

10.1186/1472-6947-4-11 Crossref , Medline ,  Google Scholar

19 Miller WL , Crabtree BF Clinical research; in Handbook of Qualitative Research , 2nd ed. Edited by Denzin NKLincoln YS Thousand Oaks, Calif, Sage, 2001 Google Scholar

20 Morse JM Principles of mixed methods and multimethod research design; in Handbook of Mixed Methods in the Social and Behavioral Sciences . Edited by , Tashakkori A , Teddlie C Thousand Oaks, Calif, Sage, 2003 Google Scholar

10.1186/1472-6963-8-156 Crossref , Medline ,  Google Scholar

22 Griswold KS , Zayas LE , Pastore PA , et al. : Primary care after psychiatric crisis: a qualitative analysis . Annals of Family Medicine 6:38–43, 2008 Crossref , Medline ,  Google Scholar

23 Trochim WM : An introduction to concept mapping for planning and evaluation . Evaluation and Program Planning 12:1–16, 1989 Crossref ,  Google Scholar

24 Aarons GA , Wells R , Zagursky K , et al. : Advancing a conceptual model of evidence-based practice implementation in child welfare . American Journal of Public Health 99:2087–2095, 2009 Crossref , Medline ,  Google Scholar

10.1186/1471-244X-7-53 Crossref ,  Google Scholar

26 Janzen R , Nelson G , Hausfather N , et al. : Capturing system level activities and impacts of mental health consumer-run organizations . American Journal of Community Health 39:287–299, 2007 Google Scholar

27 Brunette MF , Asher D , Whitley R , et al. : Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers . Psychiatric Services 59:989–995, 2008 Link ,  Google Scholar

28 Blasinsky M , Goldman HH , Unützer J : Project IMPACT: a report on barriers and facilitators to sustainability . Administration and Policy in Mental Health and Mental Health Services 33:718–729, 2006 Crossref , Medline ,  Google Scholar

29 Green CA , Polen MR , Janoff SL , et al. : Understanding how clinician-patient relationships and relational continuity of care affect recovery from serious mental illness: STARS Study results . Psychiatric Rehabilitation Journal 32:9–22, 2008 Crossref , Medline ,  Google Scholar

30 Aarons GA , Palinkas LA : Implementation of evidence-based practice in child welfare: service provider perspectives . Administration and Policy in Mental Health and Mental Health Services Research 34:411–419, 2007 Crossref , Medline ,  Google Scholar

31 Woltmann EM , Whitley R , McHugo GJ , et al. : The role of staff turnover in the implementation of evidence-based practices in health care . Psychiatric Services 59:732–737, 2008 Link ,  Google Scholar

32 Frueh BC , Cusack KJ , Grubaugh AL , et al. : Clinicians' perspectives on cognitive-behavioral treatment for PTSD among persons with severe mental illness . Psychiatric Services 57:1027–1031, 2006 Link ,  Google Scholar

33 Lucksted A , McNulty K , Brayboy L , et al. : Initial evaluation of the Peer-to-Peer Program . Psychiatric Services 60:250–253, 2009 Link ,  Google Scholar

34 Sajatovic M , Davies M , Bauer MS , et al. : Attitudes regarding the collaborative practice model and treatment adherence among individuals with bipolar disorder . Comprehensive Psychiatry 46:272–277, 2005 Crossref , Medline ,  Google Scholar

35 Proctor EK , Hascke L , Morrow-Howell N , et al. : Perceptions about competing psychosocial problems and treatment priorities among older adults with depression . Psychiatric Services 59:670–675, 2008 Link ,  Google Scholar

36 Woolfson R , Woolfson L , Mooney L , et al. : Young people's views of mental health education in secondary schools: a Scottish study . Child: Care, Health and Development 35:790–798, 2009 Crossref , Medline ,  Google Scholar

37 Proctor EK , Knudsen KL , Fedoravicius N , et al. : Implementation of evidence-based practice in community behavioral health: agency director perspectives . Administration and Policy in Mental Health and Mental Health Services Research 34:479–488, 2007 Crossref , Medline ,  Google Scholar

38 Institute of Medicine : Crossing the Quality Chasm: A New Health System for the 21st Century . Washington, DC, National Academies Press, 2000 Google Scholar

39 Mental Health : A Report of the Surgeon General . Rockville, Md, US Department of Health and Human Services, US Public Health Service, 1999 Google Scholar

10.1186/1472-6963-8-274 Crossref , Medline ,  Google Scholar

41 Gioia D , Dziadosz G : Adoption of evidence-based practices in community mental health: a mixed method study of practitioner experience . Community Mental Health Journal 44:347–357, 2008 Crossref , Medline ,  Google Scholar

42 Ostler T , Haight W , Black J , et al. : Case series: mental health needs and perspectives of rural children raised by parents who abuse methamphetamine . Journal of the American Academy of Child and Adolescent Psychiatry 46:500–507, 2007 Crossref , Medline ,  Google Scholar

10.1186/1748-3-14 Crossref , Medline ,  Google Scholar

10.1186/1472-6963-7-117 Crossref , Medline ,  Google Scholar

45 Young AT , Green CA , Estroff SE : New endeavors, risk taking, and personal growth in the recovery process: findings from the STARS study . Psychiatric Services 59:1430–1436, 2008 Link ,  Google Scholar

46 Sharkey S , MacIver S , Cameron D , et al. : An exploration of factors affecting the implementation of a randomized controlled trial of a transitional discharge model for people with serious mental illness . Journal of Psychiatric and Mental Health Nursing 12:51–56, 2005 Crossref , Medline ,  Google Scholar

47 Lee BR , Munson MR , Ware NR , et al. : Experiences of and attitudes toward mental health services among older youths in foster care . Psychiatric Services 57:487–492, 2006 Link ,  Google Scholar

48 Becker D , Whitley R , Bailey EL , et al. : Long-term employment trajectories among participants with severe mental illness in supported employment . Psychiatric Services 58:922–928, 2007 Link ,  Google Scholar

49 Wetherell JL , Ayers CR , Sorrell JT , et al. : Modular psychotherapy for anxiety in older primary care patients . American Journal of Geriatric Psychiatry 17:483–492, 2009 Crossref , Medline ,  Google Scholar

Figures and Tables

Figure 1 Common mixed-methods designs used in mental health services research

Table 1 Journals in which the 50 articles reviewed were published, with number published and 2008 impact factor

Table 2 Year of publication or of project initiation of articles and projects reviewed

Table 3 Characteristics of 50 published studies and 60 funded projects that used mixed-methods designs, by study aims

• A multi- and mixed-method adaptation study of a patient-centered perioperative mental health intervention bundle 27 October 2023 | BMC Health Services Research, Vol. 23, No. 1
• Physician Assistant Student Attitudes About People With Serious Mental Illness 21 November 2023 | Journal of Physician Assistant Education, Vol. 66
• Educators’ Perspectives on Training Mechanisms That Facilitate Evidence-Based Practice Use for Autistic Students in General Education Settings: A Mixed-Methods Analysis 2 July 2023 | Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, Vol. 46, No. 4
• Community-led identification of mental health support, challenges, and needs among Ethiopian immigrants to the U.S.: opportunities for partnership with faith communities 15 January 2024 | Mental Health, Religion & Culture, Vol. 26, No. 9
• Social network and mental health of Chinese immigrants in affordable senior housing during the COVID-19 pandemic: a mixed-methods study 22 May 2023 | Aging & Mental Health, Vol. 27, No. 10
• Incazelo nomlando oqukethwe emagameni aqanjwe abesifazane abashade ngaphambi konyaka we-1990 esigodini sakaGcaliphiwe eMaphephetheni 22 December 2023 | South African Journal of African Languages, Vol. 43, No. 3
• Implementation Science and Practice-Oriented Research: Convergence and Complementarity 30 August 2023 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 27
• Adapting to Unprecedented Times: Community Clinician Modifications to Parent–Child Interaction Therapy During COVID-19 11 August 2023 | Evidence-Based Practice in Child and Adolescent Mental Health, Vol. 8, No. 3
• Evaluating the validity of depression-related stigma measurement among diabetes and hypertension patients receiving depression care in Malawi: A mixed-methods analysis 17 May 2023 | PLOS Global Public Health, Vol. 3, No. 5
• Potential advantages of combining randomized controlled trials with qualitative research in mood and anxiety disorders - A systematic review Journal of Affective Disorders, Vol. 325
• Mental Health Therapist Perspectives on the Role of Executive Functioning in Children’s Mental Health Services 10 January 2022 | Evidence-Based Practice in Child and Adolescent Mental Health, Vol. 8, No. 1
• Therapist and supervisor perspectives about two train-the-trainer implementation strategies in schools: A qualitative study 3 August 2023 | Implementation Research and Practice, Vol. 4
• Efficacy of Therapist Guided Internet Based Cognitive Behavioural Therapy for Depression: A Qualitative Exploration of Therapists and Clients Experiences 31 December 2022 | Journal of Professional & Applied Psychology, Vol. 3, No. 4
• Prevalence of Research Designs and Efforts at Integration in Mixed Methods Research: A Systematic Review 31 December 2022 | International Journal of Multiple Research Approaches, Vol. 14, No. 3
• The measurement-based care to opioid treatment programs project (MBC2OTP): a study protocol using rapid assessment procedure informed clinical ethnography 19 August 2022 | Addiction Science & Clinical Practice, Vol. 17, No. 1
• Barbershops as a setting for supporting men's mental health during the COVID-19 pandemic: a qualitative study from the UK 27 June 2022 | BJPsych Open, Vol. 8, No. 4
• A mixed methods study of provider factors in buprenorphine treatment retention International Journal of Drug Policy, Vol. 105
• Evaluation of a systems-level technical assistance program to support youth with complex behavioral health needs Evaluation and Program Planning, Vol. 92
• Barriers to students opting-in to universities notifying emergency contacts when serious mental health concerns emerge: A UK mixed methods analysis of policy preferences Journal of Affective Disorders Reports, Vol. 7
• Development of an Online Resource for People Bereaved by Suicide: A Mixed-Method User-Centered Study Protocol 21 December 2021 | Frontiers in Psychiatry, Vol. 12
• Protocol for a hybrid type 2 cluster randomized trial of trauma-focused cognitive behavioral therapy and a pragmatic individual-level implementation strategy 7 January 2021 | Implementation Science, Vol. 16, No. 1
• Understanding adaptations in the Veteran Health Administration’s Transitions Nurse Program: refining methodology and pragmatic implications for scale-up 13 July 2021 | Implementation Science, Vol. 16, No. 1
• Defining effective care coordination for mental health referrals of refugee populations in the United States 19 November 2018 | Ethnicity & Health, Vol. 26, No. 5
• A Mixed-method Evaluation of the Behavioral Health Integration and Complex Care Initiative Using the Consolidated Framework for Implementation Research 13 May 2021 | Medical Care, Vol. 59, No. 7
• Parent Training for Youth with Autism Served in Community Settings: A Mixed-Methods Investigation Within a Community Mental Health System 2 September 2020 | Journal of Autism and Developmental Disorders, Vol. 51, No. 6
• Client, clinician, and administrator factors associated with the successful acceptance of a telehealth comprehensive recovery service: A mixed methods study Psychiatry Research, Vol. 300
• “Don’t … Break Down on Tuesday Because the Mental Health Services are Only in Town on Thursday”: A Qualitative Study of Service Provision Related Barriers to, and Facilitators of Farmers’ Mental Health Help-Seeking 15 September 2020 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 48, No. 3
• Social media and community-oriented policing: examining the organizational image construction of municipal police on Twitter and Facebook 9 November 2020 | Police Practice and Research, Vol. 22, No. 1
• The ‘shift reflection’ model of group reflective practice: a pilot study in an acute mental health setting Mental Health Practice, Vol. 24, No. 1
• Challenges Experienced by Behavioral Health Organizations in New York Resulting from COVID-19: A Qualitative Analysis 23 October 2020 | Community Mental Health Journal, Vol. 57, No. 1
• Incorporating telehealth into health service psychology training: A mixed-method study of student perspectives 24 February 2021 | DIGITAL HEALTH, Vol. 7
• An eHealth Intervention for Promoting COVID-19 Knowledge and Protective Behaviors and Reducing Pandemic Distress Among Sexual and Gender Minorities: Protocol for a Randomized Controlled Trial (#SafeHandsSafeHearts) 10 December 2021 | JMIR Research Protocols, Vol. 10, No. 12
• Promotion of mental health in young adults via mobile phone app: study protocol of the ECoWeB (emotional competence for well-being in Young adults) cohort multiple randomised trials 22 September 2020 | BMC Psychiatry, Vol. 20, No. 1
• Adaption and pilot implementation of an autism executive functioning intervention in children’s mental health services: a mixed-methods study protocol 27 April 2020 | Pilot and Feasibility Studies, Vol. 6, No. 1
• Improving the implementation and sustainment of evidence-based practices in community mental health organizations: a study protocol for a matched-pair cluster randomized pilot study of the Collaborative Organizational Approach to Selecting and Tailoring Implementation Strategies (COAST-IS) 25 February 2020 | Implementation Science Communications, Vol. 1, No. 1
• Using mixed methods in health services research: A review of the literature and case study 21 September 2020 | Journal of Health Services Research & Policy, Vol. 4
• Healthcare attendance styles among long-term unemployed people with substance-related and mood disorders Public Health, Vol. 186
• Mixed-Methods-Studien in der Gesundheitsförderung. Ergebnisse eines systematischen Reviews deutschsprachiger Publikationen Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, Vol. 153-154
• Mixed method study of workforce turnover and evidence-based treatment implementation in community behavioral health care settings Child Abuse & Neglect, Vol. 102
• Mixing Beyond Measure: Integrating Methods in a Hybrid Effectiveness–Implementation Study of Operating Room to Intensive Care Unit Handoffs 4 May 2019 | Journal of Mixed Methods Research, Vol. 14, No. 2
• The search for the ejecting chair: a mixed-methods analysis of tool use in a sedentary behavior intervention 25 November 2018 | Translational Behavioral Medicine, Vol. 10, No. 1
• SIPsmartER delivered through rural, local health districts: adoption and implementation outcomes 18 September 2019 | BMC Public Health, Vol. 19, No. 1
• An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance 10 October 2019 | Archives of Public Health, Vol. 77, No. 1
• Five Challenges in the Design and Conduct of IS Trials for HIV Prevention and Treatment JAIDS Journal of Acquired Immune Deficiency Syndromes, Vol. 82, No. 3
• Mental health recovery narratives: their impact on service users and other stakeholder groups Mental Health and Social Inclusion, Vol. 23, No. 4
• A Mixed Methods Study of Organizational Readiness for Change and Leadership During a Training Initiative Within Community Mental Health Clinics 19 June 2019 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 46, No. 5
• Associations Among Job Role, Training Type, and Staff Turnover in a Large-Scale Implementation Initiative 3 January 2019 | The Journal of Behavioral Health Services & Research, Vol. 46, No. 3
• American Journal of Community Psychology
• Internet Interventions, Vol. 18
• Journal of Public Child Welfare, Vol. 13, No. 3
• Method Sequence and Dominance in Mixed Methods Research: A Case Study of the Social Acceptance of Wind Energy Literature 12 April 2019 | International Journal of Qualitative Methods, Vol. 18
• JMIR Research Protocols, Vol. 8, No. 1
• Sundhedsprofessionelles begejstringfor fortællinger fra levet erfaring Tidsskrift for psykisk helsearbeid, Vol. 15, No. 4
• Availability of comprehensive services in permanent supportive housing in Los Angeles 6 October 2017 | Health & Social Care in the Community, Vol. 26, No. 2
• Nursing Outlook, Vol. 66, No. 2
• Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, Vol. 133
• Social Work in Mental Health, Vol. 16, No. 4
• International Journal of Family & Community Medicine, Vol. 2, No. 4
• A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives 7 December 2017 | Health Research Policy and Systems, Vol. 15, No. 1
• Fostering Psychotropic Medication Oversight for Children in Foster Care: A National Examination of States’ Monitoring Mechanisms 10 February 2016 | Administration and Policy in Mental Health and Mental Health Services Research, Vol. 44, No. 2
• Beliefs and Behaviors of Pregnant Women with Addictions Awaiting Treatment Initiation 17 November 2016 | Child and Adolescent Social Work Journal, Vol. 34, No. 1
• Psychiatric Quarterly, Vol. 88, No. 3
• Quality & Quantity, Vol. 51, No. 1
• Translational Behavioral Medicine, Vol. 7, No. 3
• Psychology, Health & Medicine, Vol. 22, No. 5
• Use of Mixed Methods Research in Research on Coronary Artery Disease, Diabetes Mellitus, and Hypertension Circulation: Cardiovascular Quality and Outcomes, Vol. 10, No. 1
• Changes in Social Networks and HIV Risk Behaviors Among Homeless Adults Transitioning Into Permanent Supportive Housing 8 July 2016 | Journal of Mixed Methods Research, Vol. 11, No. 1
• Victoria D. Ojeda , Ph.D., M.P.H. ,
• Sarah P. Hiller , M.P.I.A. ,
• Samantha Hurst , Ph.D. ,
• Nev Jones , Ph.D. ,
• Sara McMenamin , Ph.D. ,
• James Burgdorf , Ph.D. ,
• Todd P. Gilmer , Ph.D.
• Mixed-Methods Research in the Discipline of Nursing Advances in Nursing Science, Vol. 39, No. 3
• Impact of Caregiver Factors on Youth Service Utilization of Trauma-Focused Cognitive Behavioral Therapy in a Community Setting 27 February 2016 | Journal of Child and Family Studies, Vol. 25, No. 6
• Measuring Current Drug Use in Female Sex Workers and Their Noncommercial Male Partners in Mexico: Concordance Between Data Collected From Surveys Versus Semi-Structured Interviews 18 December 2015 | Substance Use & Misuse, Vol. 51, No. 1
• Administration and Policy in Mental Health and Mental Health Services Research, Vol. 43, No. 4
• Implementation and Outcomes of Forensic Housing First Programs 5 October 2015 | Community Mental Health Journal, Vol. 52, No. 1
• Academic Psychiatry, Vol. 40, No. 4
• Journal of Psychoactive Drugs, Vol. 48, No. 5
• Rural Society, Vol. 25, No. 2
• The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services 9 March 2016 | Implementation Science, Vol. 11, No. 1
• Perceptions of clinicians treating young people with first‐episode psychosis for post‐traumatic stress disorder 27 June 2013 | Early Intervention in Psychiatry, Vol. 9, No. 1
• Administration and Policy in Mental Health and Mental Health Services Research, Vol. 42, No. 2
• Administration and Policy in Mental Health and Mental Health Services Research, Vol. 42, No. 5
• Journal of Religion and Health, Vol. 54, No. 1
• The Journal of Behavioral Health Services & Research, Vol. 42, No. 4
• BMC Palliative Care, Vol. 14, No. 1
• Implementation Science, Vol. 11, No. 1
• International Journal of Environmental Research and Public Health, Vol. 12, No. 5
• Evidence-Based Programs in “Real World” Settings: Finding the Best Fit
• Ana Stefancic , M.A.
• Marian L. Katz , Ph.D.
• Marisa Sklar , M.S.
• Sam Tsemberis , Ph.D.
• Lawrence A. Palinkas , Ph.D.
• Causality and Causal Inference in Social Work 22 May 2014 | Research on Social Work Practice, Vol. 24, No. 5
• A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions 8 October 2013 | Research on Social Work Practice, Vol. 24, No. 2
• Critical Care Medicine, Vol. 42, No. 4
• Implementation Science, Vol. 9, No. 1
• Implementation Science, Vol. 8, No. 1
• Topics in research Current Opinion in Supportive & Palliative Care, Vol. 6, No. 4
• Mixed Methods for Implementation Research 5 December 2011 | Child Maltreatment, Vol. 17, No. 1
• RO1 Funding for Mixed Methods Research 2 September 2011 | Journal of Mixed Methods Research, Vol. 5, No. 4

The impact of COVID-19 on young people's mental health, wellbeing and routine from a European perspective: A co-produced qualitative systematic review

Affiliations.

• 1 NIHR Patient Safety Translational Research Centre, Institute of Global Health Innovation, Imperial College London, London, United Kingdom.
• 2 School of Public Health, Imperial College London, London, United Kingdom.
• 3 Centre for Health Policy, Institute of Global Health Innovation, Imperial College London, London, United Kingdom.
• 4 Liggins Institute, University of Auckland Waipapa Taumata Rau, Auckland, New Zealand.
• 5 Manchester Institute of Education, The University of Manchester, Manchester, United Kingdom.
• 6 School of Psychology, Liverpool John Moores University, Liverpool, United Kingdom.
• 7 Environmental Health Institute, Medicine Faculty, University of Lisbon, Lisbon, Portugal.
• 8 Newcastle Population Health Sciences Institute, Faculty of Medical Sciences, University of Newcastle, Newcastle upon Tyne, United Kingdom.
• PMID: 38507395
• PMCID: PMC10954119
• DOI: 10.1371/journal.pone.0299547

Background: The impact of the Covid-19 pandemic on young people's (YP) mental health has been mixed. Systematic reviews to date have focused predominantly on quantitative studies and lacked involvement from YP with lived experience of mental health difficulties. Therefore, our primary aim was to conduct a qualitative systematic review to examine the perceived impact of the Covid-19 pandemic on YP's (aged 10-24) mental health and wellbeing across Europe.

Methods and findings: We searched MEDLINE, PsycINFO, Embase, Web of Science, MEDRXIV, OSF preprints, Google, and voluntary sector websites for studies published from 1st January 2020 to 15th November 2022. European studies were included if they reported qualitative data that could be extracted on YP's (aged 10-24) own perspectives of their experiences of Covid-19 and related disruptions to their mental health and wellbeing. Screening, data extraction and appraisal was conducted independently in duplicate by researchers and YP with lived experience of mental health difficulties (co-researchers). Confidence was assessed using the Confidence in the Evidence from Reviews of Qualitative Research (CERQual) approach. We co-produced an adapted narrative thematic synthesis with co-researchers. This study is registered with PROSPERO, CRD42021251578. We found 82 publications and included 77 unique studies in our narrative synthesis. Most studies were from the UK (n = 50; 65%); and generated data during the first Covid-19 wave (March-May 2020; n = 33; 43%). Across the 79,491 participants, views, and experiences of YP minoritised by ethnicity and sexual orientation, and from marginalised or vulnerable YP were limited. Five synthesised themes were identified: negative impact of pandemic information and restrictions on wellbeing; education and learning on wellbeing; social connection to prevent loneliness and disconnection; emotional, lifestyle and behavioural changes; and mental health support. YP's mental health and wellbeing across Europe were reported to have fluctuated during the pandemic. Challenges were similar but coping strategies to manage the impact of these challenges on mental health varied across person, study, and country. Short-term impacts were related to the consequences of changing restrictions on social connection, day-to-day lifestyle, and education set-up. However, YP identified potential issues in these areas going forward, and therefore stressed the importance of ongoing long-term support in education, learning and mental health post-Covid-19.

Conclusions: Our findings map onto the complex picture seen from quantitative systematic reviews regarding the impact of Covid-19 on YP's mental health. The comparatively little qualitative data found in our review means there is an urgent need for more high-quality qualitative research outside of the UK and/or about the experiences of minoritised groups to ensure all voices are heard and everyone is getting the support they need following the pandemic. YP's voices need to be prioritised in decision-making processes on education, self-care strategies, and mental health and wellbeing, to drive impactful, meaningful policy changes in anticipation of a future systemic crisis.

Copyright: © 2024 Dewa et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Publication types

• COVID-19* / epidemiology
• Mental Health*
• Qualitative Research

Grants and funding

• Systematic review
• Open access
• Published: 10 October 2019

An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance

• Eric Badu   ORCID: orcid.org/0000-0002-0593-3550 1 ,
• Anthony Paul O’Brien 2 &
• Rebecca Mitchell 3  

Archives of Public Health volume  77 , Article number:  37 ( 2019 ) Cite this article

21k Accesses

19 Citations

1 Altmetric

Metrics details

Several typologies and guidelines are available to address the methodological and practical considerations required in mental health research. However, few studies have actually attempted to systematically identify and synthesise these considerations. This paper provides an integrative review that identifies and synthesises the available research evidence on mental health research methodological considerations.

A search of the published literature was conducted using EMBASE, Medline, PsycINFO, CINAHL, Web of Science, and Scopus. The search was limited to papers published in English for the timeframe 2000–2018. Using pre-defined inclusion and exclusion criteria, three reviewers independently screened the retrieved papers. A data extraction form was used to extract data from the included papers.

Of 27 papers meeting the inclusion criteria, 13 focused on qualitative research, 8 mixed methods and 6 papers focused on quantitative methodology. A total of 14 papers targeted global mental health research, with 2 papers each describing studies in Germany, Sweden and China. The review identified several methodological considerations relating to study design, methods, data collection, and quality assurance. Methodological issues regarding the study design included assembling team members, familiarisation and sharing information on the topic, and seeking the contribution of team members. Methodological considerations to facilitate data collection involved adequate preparation prior to fieldwork, appropriateness and adequacy of the sampling and data collection approach, selection of consumers, the social or cultural context, practical and organisational skills; and ethical and sensitivity issues.

The evidence confirms that studies on methodological considerations in conducting mental health research largely focus on qualitative studies in a transcultural setting, as well as recommendations derived from multi-site surveys. Mental health research should adequately consider the methodological issues around study design, sampling, data collection procedures and quality assurance in order to maintain the quality of data collection.

Peer Review reports

In the past decades there has been considerable attention on research methods to facilitate studies in various academic fields, such as public health, education, humanities, behavioural and social sciences [ 1 , 2 , 3 , 4 ]. These research methodologies have generally focused on the two major research pillars known as quantitative or qualitative research. In recent years, researchers conducting mental health research appear to be either employing both qualitative and quantitative research methods separately, or mixed methods approaches to triangulate and validate findings [ 5 , 6 ].

A combination of study designs has been utilised to answer research questions associated with mental health services and consumer outcomes [ 7 , 8 ]. Study designs in the public health and clinical domains, for example, have largely focused on observational studies (non-interventional) and experimental research (interventional) [ 1 , 3 , 9 ]. Observational design in non-interventional research requires the investigator to simply observe, record, classify, count and analyse the data [ 1 , 2 , 10 ]. This design is different from the observational approaches used in social science research, which may involve observing (participant and non- participant) phenomena in the fieldwork [ 1 ]. Furthermore, the observational study has been categorized into five types, namely cross-sectional design, case-control studies, cohort studies, case report and case series studies [ 1 , 2 , 3 , 9 , 10 , 11 ]. The cross-sectional design is used to measure the occurrence of a condition at a one-time point, sometimes referred to as a prevalence study. This approach of conducting research is relatively quick and easy but does not permit a distinction between cause and effect [ 1 ]. Conversely, the case-control is a design that examines the relationship between an attribute and a disease by comparing those with and without the disease [ 1 , 2 , 12 ]. In addition, the case-control design is usually retrospective and aims to identify predictors of a particular outcome. This type of design is relevant when investigating rare or chronic diseases which may result from long-term exposure to particular risk factors [ 10 ]. Cohort studies measure the relationship between exposure to a factor and the probability of the occurrence of a disease [ 1 , 10 ]. In a case series design, medical records are reviewed for exposure to determinants of disease and outcomes. More importantly, case series and case reports are often used as preliminary research to provide information on key clinical issues [ 12 ].

The interventional study design describes a research approach that applies clinical care to evaluate treatment effects on outcomes [ 13 ]. Several previous studies have explained the various forms of experimental study design used in public health and clinical research [ 14 , 15 ]. In particular, experimental studies have been categorized into randomized controlled trials (RCTs), non-randomized controlled trials, and quasi-experimental designs [ 14 ]. The randomized trial is a comparative study where participants are randomly assigned to one of two groups. This research examines a comparison between a group receiving treatment and a control group receiving treatment as usual or receiving a placebo. Herein, the exposure to the intervention is determined by random allocation [ 16 , 17 ].

Recently, research methodologists have given considerable attention to the development of methodologies to conduct research in vulnerable populations. Vulnerable population research, such as with mental health consumers often involves considering the challenges associated with sampling (selecting marginalized participants), collecting data and analysing it, as well as research engagement. Consequently, several empirical studies have been undertaken to document the methodological issues and challenges in research involving marginalized populations. In particular, these studies largely addresses the typologies and practical guidelines for conducting empirical studies in mental health. Despite the increasing evidence, however, only a few studies have yet attempted to systematically identify and synthesise the methodological considerations in conducting mental health research from the perspective of consumers.

A preliminary search using the search engines Medline, Web of Science, Google Scholar, and Scopus Index and EMBASE identified only two reviews of mental health based research. Among these two papers, one focused on the various types of mixed methods used in mental health research [ 18 ], whilst the other paper, focused on the role of qualitative studies in mental health research involving mixed methods [ 19 ]. Even though the latter two studies attempted to systematically review mixed methods mental health research, this integrative review is unique, as it collectively synthesises the design, data collection, sampling, and quality assurance issues together, which has not been previously attempted.

This paper provides an integrative review addressing the available evidence on mental health research methodological considerations. The paper also synthesises evidence on the methods, study designs, data collection procedures, analyses and quality assurance measures. Identifying and synthesising evidence on the conduct of mental health research has relevance to clinicians and academic researchers where the evidence provides a guide regarding the methodological issues involved when conducting research in the mental health domain. Additionally, the synthesis can inform clinicians and academia about the gaps in the literature related to methodological considerations.

Methodology

An integrative review was conducted to synthesise the available evidence on mental health research methodological considerations. To guide the review, the World Health Organization (WHO) definition of mental health has been utilised. The WHO defines mental health as: “a state of well-being, in which the individual realises his or her own potentials, ability to cope with the normal stresses of life, functionality and work productivity, as well as the ability to contribute effectively in community life” [ 20 ]. The integrative review enabled the simultaneous inclusion of diverse methodologies (i.e., experimental and non-experimental research) and varied perspectives to fully understand a phenomenon of concern [ 21 , 22 ]. The review also uses diverse data sources to develop a holistic understanding of methodological considerations in mental health research. The methodology employed involves five stages: 1) problem identification (ensuring that the research question and purpose are clearly defined); 2) literature search (incorporating a comprehensive search strategy); 3) data evaluation; 4) data analysis (data reduction, display, comparison and conclusions) and; 5) presentation (synthesising findings in a model or theory and describing the implications for practice, policy and further research) [ 21 ].

Inclusion criteria

The integrative review focused on methodological issues in mental health research. This included core areas such as study design and methods, particularly qualitative, quantitative or both. The review targeted papers that addressed study design, sampling, data collection procedures, quality assurance and the data analysis process. More specifically, the included papers addressed methodological issues on empirical studies in mental health research. The methodological issues in this context are not limited to a particular mental illness. Studies that met the inclusion criteria were peer-reviewed articles published in the English Language, from January 2000 to July 2018.

Exclusion criteria

Articles that were excluded were based purely on general health services or clinical effectiveness of a particular intervention with no connection to mental health research. Articles were also excluded when it addresses non-methodological issues. Other general exclusion criteria were book chapters, conference abstracts, papers that present opinion, editorials, commentaries and clinical case reviews.

Search strategy and selection procedure

The search of published articles was conducted from six electronic databases, namely EMBASE, CINAHL (EBSCO), Web of Science, Scopus, PsycINFO and Medline. We developed a search strategy based on the recommended guidelines by the Joanna Briggs Institute (JBI) [ 23 ]. Specifically, a three-step search strategy was utilised to conduct the search for information (see Table  1 ). An initial limited search was conducted in Medline and Embase (see Table 1 ). We analysed the text words contained in the title and abstract and of the index terms from the initial search results [ 23 ]. A second search using all identified keywords and index terms was then repeated across all remaining five databases (see Table 1 ). Finally, the reference lists of all eligible studies were manually hand searched [ 23 ].

The selection of eligible articles adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 24 ] (see Fig.  1 ). Firstly, three authors independently screened the titles of articles that were retrieved and then approved those meeting the selection criteria. The authors reviewed all the titles and abstracts and agreed on those needing full-text screening. E.B (Eric Badu) conducted the initial screening of titles and abstracts. A.P.O’B (Anthony Paul O’Brien) and R.M (Rebecca Mitchell) conducted the second screening of titles and abstracts of all the identified papers. The authors (E.B, A.P.O’B and R.M) conducted full-text screening according to the inclusion and exclusion criteria.

Flow Chart of studies included in the review

Data management and extraction

The integrative review used Endnote ×8 to screen and handle duplicate references. A predefined data extraction form was developed to extract data from all included articles (see Additional file 1 ). The data extraction form was developed according to Joanna Briggs Institute (JBI) [ 23 ] and Cochrane [ 24 ] manuals, as well as the literature associated with concepts and methods in mental health research. The data extraction form was categorised into sub-sections, such as study details (citation, year of publication, author, contact details of lead author, and funder/sponsoring organisation, publication type), objective of the paper, primary subject area of the paper (study design, methods, sampling, data collection, data analysis, quality assurance). The data extraction form also had a section on additional information on methodological consideration, recommendations and other potential references. The authors extracted results of the included papers in numerical and textual format [ 23 ]. EB (Eric Badu) conducted the data extraction, A.P.O’B (Anthony Paul O’Brien) and R.M (Rebecca Mitchell), conducted the second review of the extracted data.

Data synthesis

Content analysis was used to synthesise the extracted data. The content analysis process involved several stages which involved noting patterns and themes, seeing plausibility, clustering, counting, making contrasts and comparisons, discerning common and unusual patterns, subsuming particulars into general, noting relations between variability, finding intervening factors and building a logical chain of evidence [ 21 ] (see Table  2 ).

Study characteristics

The integrative review identified a total of 491 records from all databases, after which 19 duplicates were removed. Out of this, 472 titles and abstracts were assessed for eligibility, after which 439 articles were excluded. Articles not meeting the inclusion criteria were excluded. Specifically, papers excluded were those that did not address methodological issues as well as papers addressing methodological consideration in other disciplines. A total of 33 full-text articles were assessed – 9 articles were further excluded, whilst an additional 3 articles were identified from reference lists. Overall, 27 articles were included in the final synthesis (see Fig. 1 ). Of the total included papers, 12 contained qualitative research, 9 were mixed methods (both qualitative and quantitative) and 6 papers focused on quantitative data. Conversely, a total of 14 papers targeted global mental health research and 2 papers each describing studies in Germany, Sweden and China. The papers addressed different methodological issues, such as study design, methods, data collection, and analysis as well as quality assurance (see Table  3 ).

Mixed methods design in mental health research

Mixed methods research is defined as a research process where the elements of qualitative and quantitative research are combined in the design, data collection, and its triangulation and validation [ 48 ]. The integrative review identified four sub-themes that describe mixed methods design in the context of mental health research. The sub-themes include the categories of mixed methods, their function, structure, process and further methodological considerations for mixed methods design. These sub-themes are explained as follows:

Categorizing mixed methods in mental health research

Four studies highlighted the categories of mixed methods design applicable to mental health research [ 18 , 19 , 43 , 48 ]. Generally, there are differences in the categories of mixed methods design, however, three distinct categories predominantly appear to cross cut in all studies. These categories are function, structure and process. Some studies further categorised mixed method design to include rationale, objectives, or purpose. For instance, Schoonenboom and Johnson [ 48 ] categorised mixed methods design into primary and secondary dimensions.

The function of mixed methods in mental health research

Six studies explain the function of conducting mixed methods design in mental health research. Two studies specifically recommended that mixed methods have the ability to provide a more robust understanding of services by expanding and strengthening the conclusions from the study [ 42 , 45 ]. More importantly, the use of both qualitative and quantitative methods have the ability to provide innovative solutions to important and complex problems, especially by addressing diversity and divergence [ 48 ]. The review identified five underlying functions of a mixed method design in mental health research which include achieving convergence, complementarity, expansion, development and sampling [ 18 , 19 , 43 ].

The use of mixed methods to achieve convergence aims to employ both qualitative and quantitative data to answer the same question, either through triangulation (to confirm the conclusions from each of the methods) or transformation (using qualitative techniques to transform quantitative data). Similarly, complementarity in mixed methods integrates both qualitative and quantitative methods to answer questions for the purpose of evaluation or elaboration [ 18 , 19 , 43 ]. Two papers recommend that qualitative methods are used to provide the depth of understanding, whilst the quantitative methods provide a breadth of understanding [ 18 , 43 ]. In mental health research, the qualitative data is often used to examine treatment processes, whilst the quantitative methods are used to examine treatment outcomes against quality care key performance targets.

Additionally, three papers indicated that expansion as a function of mixed methods uses one type of method to answer questions raised by the other type of method [ 18 , 19 , 43 ]. For instance, qualitative data is used to explain findings from quantitative analysis. Also, some studies highlight that development as a function of mixed methods aims to use one method to answer research questions, and use the findings to inform other methods to answer different research questions. A qualitative method, for example, is used to identify the content of items to be used in a quantitative study. This approach aims to use qualitative methods to create a conceptual framework for generating hypotheses to be tested by using a quantitative method [ 18 , 19 , 43 ]. Three papers suggested that using mixed methods for the purpose of sampling utilize one method (eg. quantitative) to identify a sample of participants to conduct research using other methods (eg. qualitative) [ 18 , 19 , 43 ]. For instance, quantitative data is sequentially utilized to identify potential participants to participate in a qualitative study and the vice versa.

Structure of mixed methods in mental health research

Five studies categorised the structure of conducting mixed methods in mental health research, into two broader concepts including simultaneous (concurrent) and sequential (see Table 3 ). In both categories, one method is regarded as primary and the other as secondary, although equal weight can be given to both methods [ 18 , 19 , 42 , 43 , 48 ]. Two studies suggested that the sequential design is a process where the data collection and analysis of one component (eg. quantitative) takes place after the data collection and analysis of the other component (eg qualitative). Herein, the data collection and analysis of one component (e.g. qualitative) may depend on the outcomes of the other component (e.g. quantitative) [ 43 , 48 ]. An earlier review suggested that the majority of contemporary studies in mental health research use a sequential design, with qualitative methods, more often preceding quantitative methods [ 18 ].

Alternatively, the concurrent design collects and analyses data of both components (e.g. quantitative and qualitative) simultaneously and independently. Palinkas, Horwitz [ 42 ] recommend that one component is used as secondary to the other component, or that both components are assigned equal priority. Such a mixed methods approach aims to provide a depth of understanding afforded by qualitative methods, with the breadth of understanding offered by the quantitative data to elaborate on the findings of one component or seek convergence through triangulation of the results. Schoonenboom and Johnson [ 48 ] recommended the use of capital letters for one component and lower case letters for another component in the same design to indicate that one component is primary and the other is secondary or supplemental.

Process of mixed methods in mental health research

Five papers highlighted the process for the use of mixed methods in mental health research [ 18 , 19 , 42 , 43 , 48 ]. The papers suggested three distinct processes or strategies for combining qualitative and quantitative data. These include merging or converging the two data sets, connecting the two datasets by having one build upon the other; and embedding one data set within the other [ 19 , 43 ]. The process of connecting occurs when the analysis of one dataset leads to the need for the other data set. For instance, in the situation where quantitative results lead to the subsequent collection and analysis of qualitative data [ 18 , 43 ]. A previous study suggested that most studies in mental health sought to connect the data sets. Similarly, the process of merging the datasets brings together two sets of data during the interpretation, or transforms one type of data into the other type, by combining the data into new variables [ 18 ]. The process of embedding data into mixed method designs in mental health uses one dataset to provide a supportive role to the other dataset [ 43 ].

Consideration for using mixed methods in mental health research

Three studies highlighted several factors that need to be considered when conducting mixed methods design in mental health research [ 18 , 19 , 45 ]. Accordingly, these factors include developing familiarity with the topic under investigation based on experience, willingness to share information on the topic [ 19 ], establishing early collaboration, willingness to negotiate emerging problems, seeking the contribution of team members, and soliciting third-party assistance to resolve any emerging problems [ 45 ]. Additionally, Palinkas, Horwitz [ 18 ] recommended that mixed methods in the context of mental health research are mostly applied in studies that assess needs of services, examine existing services, developing new or adapting existing services, evaluating services in randomised control trials, and examining service implementation.

Qualitative study in mental health research

This theme describes the various qualitative methods used in mental health research. The theme also addresses methodological considerations for using qualitative methods in mental health research. The key emerging issues are discussed below:

Considering qualitative components in conducting mental health research

Six studies recommended the use of qualitative methods in mental health research [ 19 , 26 , 28 , 32 , 36 , 44 ]. Two qualitative research paradigms were identified, including the interpretive and critical approach [ 32 ]. The interpretive methodologies predominantly explore the meaning of human experiences and actions, whilst the critical approach emphasises the social and historical origins and contexts of meaning [ 32 ]. Two studies suggested that the interpretive qualitative methods used in mental health research are ethnography, phenomenology and narrative approaches [ 32 , 36 ].

The ethnographic approach describes the everyday meaning of the phenomena within a societal and cultural context, for instance, the way phenomena or experience is contrasted within a community, or by collective members over time [ 32 ]. Alternatively, the phenomenological approach explores the claims and concerns of a subject with a speculative development of an interpretative account within their cultural and physical environments focusing on the lived experience [ 32 , 36 ].

Moreover, the critical qualitative approaches used in mental health research are predominantly emancipatory (for instance, socio-political traditions) and participatory action-based research. The emancipatory traditions recognise that knowledge is acquired through critical discourse and debate but are not seen as discovered by objective inquiry [ 32 ]. Alternatively, the participatory action based approach uses critical perspectives to engage key stakeholders as participants in the design and conduct of the research [ 32 ].

Some studies highlighted several reasons why qualitative methods are relevant to mental health research. In particular, qualitative methods are significant as they emphasise naturalistic inquiry and have a discovery-oriented approach [ 19 , 26 ]. Two studies suggested that qualitative methods are often relevant in the initial stages of research studies to understand specific issues such as behaviour, or symptoms of consumers of mental services [ 19 ]. Specifically, Palinkas [ 19 ] suggests that qualitative methods help to obtain initial pilot data, or when there is too little previous research or in the absence of a theory, such as provided in exploratory studies, or previously under-researched phenomena.

Three studies stressed that qualitative methods can help to better understand socially sensitive issues, such as exploring the solutions to overcome challenges in mental health clinical policies [ 19 , 28 , 44 ]. Consequently, Razafsha, Behforuzi [ 44 ] recommended that the natural holistic view of qualitative methods can help to understand the more recovery-oriented policy of mental health, rather than simply the treatment of symptoms. Similarly, the subjective experiences of consumers using qualitative approaches have been found useful to inform clinical policy development [ 28 ].

Sampling in mental health research

The theme explains the sampling approaches used in mental health research. The section also describes the methodological considerations when sampling participants for mental health research. The sub-themes emerging are explained in the following sections:

Sampling approaches (quantitative)

Some studies reviewed highlighted the sampling approaches previously used in mental health research [ 25 , 34 , 35 ]. Generally, all quantitative studies tend to use several probability sampling approaches, whilst qualitative studies used non-probability techniques. The quantitative mental health studies conducted at community and population level employ multi-stage sampling techniques usually involving systematic sampling, stratified and random sampling [ 25 , 34 ]. Similarly, quantitative studies that recruit consumers in the hospital setting employ consecutive sampling [ 35 ]. Two studies reviewed highlighted that the identification of consumers of mental health services for research is usually conducted by service providers. For instance, Korver, Quee [ 35 ] research used a consecutive sampling approach by identifying consumers through clinicians working in regional psychosis departments, or academic centres.

Sampling approaches (qualitative)

Seven studies suggested that the sampling procedures widely used in mental health research involving qualitative methods are non-probability techniques, which include purposive [ 19 , 28 , 32 , 42 , 46 ], snowballing [ 30 , 32 , 46 ] and theoretical sampling [ 31 , 32 ]. The purposive sampling identifies participants that possess relevant characteristics to answer a research question [ 28 ]. Purposive sampling can be used in a single case study, or for multiple cases. The purposive sampling used in mental health research is usually extreme, or deviant case sampling, criterion sampling, and maximum variation sampling [ 19 ]. Furthermore, it is advised when using purposive sampling in a multistage level study, that it should aim to begin with the broader picture to achieve variation, or dispersion, before moving to the more focused view that considers similarity, or central tendencies [ 42 ].

Two studies added that theoretical sampling involved sampling participants, situations and processes based on concepts on theoretical grounds and then using the findings to build theory, such as in a Grounded Theory study [ 31 , 32 ]. Some studies highlighted that snowball sampling is another strategy widely used in mental health research [ 30 , 32 , 46 ]. This is ascribed to the fact that people with mental illness are perceived as marginalised in research and practically hard-to-reach using conventional sampling [ 30 , 32 ]. Snowballing sampling involves asking the marginalised participants to recommend individuals who might have direct knowledge relevant to the study [ 30 , 32 , 46 ]. Although this approach is relevant, some studies advise the limited possibility of generalising the sample, because of the likelihood of selection bias [ 30 ].

Sampling consideration

Four studies in this section highlighted some of the sampling considerations in mental health research [ 30 , 31 , 32 , 46 ]. Generally, mental health research should consider the appropriateness and adequacy of sampling approach by applying attributes such as shared social, or cultural experiences, or shared concern related to the study [ 32 ], diversity and variety of participants [ 31 ], practical and organisational skills, as well as ethical and sensitivity issues [ 46 ]. Robinson [ 46 ] further suggested that sampling can be homogenous or heterogeneous depending on the research questions for the study. Achieving homogeneity in sampling should employ a variety of parameters, which include demographic, graphical, physical, psychological, or life history homogeneity [ 46 ]. Additionally, applying homogeneity in sampling can be influenced by theoretical and practical factors. Alternatively, some samples are intentionally selected based on heterogeneous factors [ 46 ].

Data collection in mental health research

This theme highlights the data collection methods used in mental health research. The theme is explained according to three sub-themes, which include approaches for collecting qualitative data, methodological considerations, as well as preparations for data collection. The sub-themes are as follows:

Approaches for collecting qualitative data

The studies reviewed recommended the approaches that are widely applied in collecting data in mental health research. The widely used qualitative data collection approaches in mental health research are focus group discussions (FGDs) [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ], extended in-depth interviews [ 19 , 30 , 34 ], participant and non-participant observation [ 19 ], Delphi data collection, quasi-statistical techniques [ 19 ] and field notes [ 31 , 40 ]. Seven studies suggest that FGDs are widely used data collection approaches [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ] because they are valuable in gathering information on consumers’ perspectives of services, especially regarding satisfaction, unmet/met service needs and the perceived impact of services [ 47 ]. Conversely, Ekblad and Baarnhielm [ 31 ] recommended that this approach is relevant to improve clinical understanding of the thoughts, emotions, meanings and attitudes towards mental health services.

Such data collection approaches are particularly relevant to consumers of mental health services, due to their low self-confidence and self-esteem [ 41 ]. The approach can help to understand specific terms, vocabulary, opinions and attitudes of consumers of mental health services, as well as their reasoning about personal distress and healing [ 31 ]. Similarly, the reliance on verbal rather than written communication helps to promote the participation of participants with serious and enduring mental health problems [ 31 , 41 ]. Although FGD has several important outcomes, there are some limitations that need critical consideration. Ekblad and Baarnhielm [ 31 ] for example suggest, that marginalised participants may not always feel free to talk about private issues regarding their condition at the group level mostly due to perceived stigma and group confidentiality.

Some studies reviewed recommended that attempting to capture comprehensive information and analysing group interactions in mental health research requires the research method to use field notes as a supplementary data source to help validate the FGDs [ 31 , 40 , 41 ]. The use of field notes in addition to FGDs essentially provides greater detail in the accounts of consumers’ subjective experiences. Furthermore, Montgomery and Bailey [ 40 ] suggest that field notes require observational sensitivity, and also require having specific content such as descriptive and interpretive data.

Three studies in this section suggested that in-depth interviews are used to collect data from consumers of mental health services [ 19 , 30 , 34 ]. This approach is particularly important to explore the behaviour, subjective experiences and psychological processes; opinions, and perceptions of mental health services. de Jong and Van Ommeren [ 30 ] recommend that in-depth interviews help to collect data on culturally marked disorders, their personal and interpersonal significance, patient and family explanatory models, individual and family coping styles, symptom symbols and protective mediators. Palinkas [ 19 ] also highlights that the structured narrative form of extended interviewing is the type of in-depth interview used in mental health research. This approach provides participants with the opportunity to describe the experience of living with an illness and seeking services that assist them.

Consideration for data collection

Six studies recommended consideration required in the data collection process [ 31 , 32 , 37 , 41 , 47 , 49 ]. Some studies highlighted that consumers of mental health services might refuse to participate in research due to several factors [ 37 ] like the severity of their illness, stigma and discrimination [ 41 ]. Subsequently, such issues are recommended to be addressed by building confidence and trust between the researcher and consumers [ 31 , 37 ]. This is a significant prerequisite, as it can sensitise and normalise the research process and aims with the participants prior to discussing their personal mental health issues. Similarly, some studies added that the researcher can gain the confidence of service providers who manage consumers of mental health services [ 41 , 47 ], seek ethical approval from the relevant committee(s) [ 41 , 47 ], meet and greet the consumers of mental health services before data collection, and arrange a mutually acceptable venue for the groups and possibly supply transport [ 41 ].

Two studies further suggested that the cultural and social differences of the participants need consideration [ 26 , 31 ]. These factors could influence the perception and interpretation of ethical issues in the research situation.

Additionally, two studies recommended the use of standardised assessment instruments for mental health research that involve quantitative data collection [ 33 , 49 ]. A recent survey suggested that measures to standardise the data collection approach can convert self-completion instruments to interviewer-completion instruments [ 49 ]. The interviewer can then read the items of the instruments to respondents and record their responses. The study further suggested the need to collect demographic and behavioural information about the participant(s).

Preparing for data collection

Eight studies highlighted the procedures involved in preparing for data collection in mental health research [ 25 , 30 , 33 , 34 , 35 , 39 , 41 , 49 ]. These studies suggest that the preparation process involve organising meetings of researchers, colleagues and representatives of the research population. The meeting of researchers generally involves training of interviewers about the overall design, objectives and research questions associated with the study. de Jong and Van Ommeren [ 30 ] recommended that preparation for the use of quantitative data encompasses translating and adapting instruments with the aim of achieving content, semantic, concept, criterion and technical equivalence.

Quality assurance procedures in mental health research

This section describes the quality assurance procedures used in mental health research. Quality assurance is explained according to three sub-themes: 1) seeking informed consent, 2) the procedure for ensuring quality assurance in a quantitative study and 3) the procedure for ensuring quality control in a qualitative study. The sub-themes are explained in the following content.

Seeking informed consent

The papers analysed for the integrative review suggested that the rights of participants to safeguard their integrity must always be respected, and so each potential subject must be adequately informed of the aims, methods, anticipated benefits and potential hazards of the study and any potential discomforts (see Table 3 ). Seven studies highlight that potential participants of mental health research must be consented to the study prior to data collection [ 25 , 26 , 33 , 35 , 37 , 39 , 47 ]. The consent process helps to assure participants of anonymity and confidentiality and further explain the research procedure to them. Baarnhielm and Ekblad [ 26 ] argue that the research should be guided by four basic moral values for medical ethics, autonomy, non-maleficence, beneficence, and justice. In particular, potential consumers of mental health services who may have severe conditions and unable to consent themselves are expected to have their consent signed by a respective family caregiver [ 37 ]. Latvala, Vuokila-Oikkonen [ 37 ] further suggested that researchers are responsible to agree on the criteria to determine the competency of potential participants in mental health research. The criteria are particularly relevant when potential participants have difficulties in understanding information due to their mental illness.

Procedure for ensuring quality control (quantitative)

Several studies highlighted procedures for ensuring quality control in mental health research (see Table 3 ). The quality control measures are used to achieve the highest reliability, validity and timeliness. Some studies demonstrate that ensuring quality control should consider factors such as pre-testing tools [ 25 , 49 ], minimising non-response rates [ 25 , 39 ] and monitoring of data collection processes [ 25 , 33 , 49 ].

Accordingly, two studies suggested that efforts should be made to re-approach participants who initially refuse to participate in the study. For instance, Liu, Huang [ 39 ] recommended that when a consumer of mental health services refuse to participate in a study (due to low self-esteem) when approached for the first time, a different interviewer can re-approach the same participant to see if they are more comfortable to participate after the first invitation. Three studies further recommend that monitoring data quality can be accomplished through “checks across individuals, completion status and checks across variables” [ 25 , 33 , 49 ]. For example, Alonso, Angermeyer [ 25 ] advocate that various checks are used to verify completion of the interview, and consistency across instruments against the standard procedure.

Procedure for ensuring quality control (qualitative)

Four studies highlighted the procedures for ensuring quality control of qualitative data in mental health research [ 19 , 32 , 37 , 46 ]. A further two studies suggested that the quality of qualitative research is governed by the principles of credibility, dependability, transferability, reflexivity, confirmability [ 19 , 32 ]. Some studies explain that the credibility or trustworthiness of qualitative research in mental health is determined by methodological and interpretive rigour of the phenomenon being investigated [ 32 , 37 ]. Consequently, Fossey, Harvey [ 32 ] propose that the methodological rigour for assessing the credibility of qualitative research are congruence, responsiveness or sensitivity to social context, appropriateness (importance and impact), adequacy and transparency. Similarly, interpretive rigour is classified as authenticity, coherence, reciprocity, typicality and permeability of the researcher’s intentions; including engagement and interpretation [ 32 ].

Robinson [ 46 ] explained that transparency (openness and honesty) is achieved if the research report explicitly addresses how the sampling, data collection, analysis, and presentation are met. In particular, efforts to address these methodological issues highlight the extent to which the criteria for quality profoundly interacts with standards for ethics. Similarly, responsiveness, or sensitivity, helps to situate or locate the study within a place, a time and a meaningful group [ 46 ]. The study should also consider the researcher’s background, location and connection to the study setting, particularly in the recruitment process. This is often described as role conflict or research bias.

In the interpretive phenomenon, coherence highlights the ability to select an appropriate sampling procedure that mutually matches the research aims, questions, data collection, analysis, as well as any theoretical concepts or frameworks [ 32 , 46 ]. Similarly, authenticity explains the appropriate representation of participants’ perspectives in the research process and the interpretation of results. Authenticity is maximised by providing evidence that participants are adequately represented in the interpretive process, or provided an opportunity to give feedback on the researcher’s interpretation [ 32 ]. Again, the contribution of the researcher’s perspective to the interpretation enhances permeability. Fossey, Harvey [ 32 ] further suggest that reflexive reporting, which distinguishes the participants’ voices from that of the researcher in the report, enhances the permeability of the researcher’s role and perspective.

One study highlighted the approaches used to ensure validity in qualitative research, which includes saturation, identification of deviant or non-confirmatory cases, member checking and coding by consensus. Saturation involves completeness in the research process, where all relevant data collection, codes and themes required to answer the phenomenon of inquiry are achieved; and no new data emerges [ 19 ]. Similarly, member checking is the process whereby participants or others who share similar characteristics review study findings to elaborate on confirming them [ 19 ]. The coding by consensus involves a collaborative approach to analysing the data. Ensuring regular meetings among coders to discuss procedures for assigning codes to segments of data and resolve differences in coding procedures, and by comparison of codes assigned on selected transcripts to calculate a percentage agreement or kappa measure of interrater reliability, are commonly applied [ 19 ].

Two studies recommend the need to acknowledge the importance of generalisability (transferability). This concept aims to provide sufficient information about the research setting, findings and interpretations for readers to appropriately determine the replicability of the findings from one context, or population to another, otherwise known as reliability in quantitative research [ 19 , 32 ]. Similarly, the researchers should employ reflexivity as a means of identifying and addressing potential biases in data collection and interpretation. Palinkas [ 19 ] suggests that such bias is associated with theoretical orientations; pre-conceived beliefs, assumptions, and demographic characteristics; and familiarity and experience with the methods and phenomenon. Another approach to enhance the rigour of analysis involves peer debriefing and support meetings held among team members which facilitate detailed auditing during data analysis [ 19 ].

The integrative review was conducted to synthesise evidence into recommended methodological considerations when conducting mental health research. The evidence from the review has been discussed according to five major themes: 1) mixed methods study in mental health research; 2) qualitative study in mental health research; 3) sampling in mental health research; 4) data collection in mental health research; and 5) quality assurance procedures in mental health research.

Mixed methods study in mental health research

The evidence suggests that mixed methods approach in mental health are generally categorised according to their function (rationale, objectives or purpose), structure and process [ 18 , 19 , 43 , 48 ]. The mixed methods study can be conducted for the purpose of achieving convergence, complementarity, expansion, development and sampling [ 18 , 19 , 43 ]. Researchers conducting mental health studies should understand the underlying functions or purpose of mixed methods. Similarly, mixed methods in mental health studies can be structured simultaneously (concurrent) and sequential [ 18 , 19 , 42 , 43 , 48 ]. More importantly, the process of combining qualitative and quantitative data can be achieved through merging or converging, connecting and embedding one data set within the other [ 18 , 19 , 42 , 43 , 48 ]. The evidence further recommends that researchers need to understand the stage of integrating the two sets of data and the rationale for doing so. This can inform researchers regarding the best stage and appropriate ways of combining the two components of data to adequately address the research question(s).

The evidence recommended some methodological consideration in the design of mixed methods projects in mental health [ 18 , 19 , 45 ]. These issues include establishing early collaboration, becoming familiar with the topic, sharing information on the topic, negotiating any emerging problems and seeking contributions from team members. The involvement of various expertise could ensure that methodological issues are clearly identified. However, addressing such issues midway, or late through the design can negatively affect the implementation [ 45 ]. Any robust discoveries can rarely be accommodated under the existing design. Therefore, the inclusion of various methodological expertise during inception can lead to a more robust mixed-methods design which maximises the contributions of team members. Whilst fundamental and philosophical differences in qualitative and quantitative methods may not be resolved, some workable solutions can be employed, particularly if challenges are viewed as philosophical rather than personal [ 45 ]. The cultural issues can be alleviated by understanding the concepts, norms and values of the setting, further to respecting and including perspectives of the various stakeholders.

The review findings suggest that qualitative methods are relevant when conducting mental health research. The qualitative methods are mostly used where there has been limited previous research and an absence of theoretical perspectives. The approach is also used to gather initial pilot data. More importantly, the qualitative methods are relevant when we want to understand sensitive issues, especially from consumers of mental health services, where the ‘lived experience is paramount [ 19 , 28 , 44 ]. Qualitative methods can help understand the experiences of consumers in the process of treatment, as well as their therapeutic relationship with mental health professionals. The experiences of consumers from qualitative data are particularly important in developing clinical policy [ 28 ]. The review findings find two paradigms of qualitative methods are used in mental health research. These paradigms are the interpretive and critical approach [ 32 ]. The interpretive qualitative method(s) include phenomenology, ethnography and narrative approaches [ 32 , 36 ]. Conversely, critical qualitative approaches are participatory action research and emancipatory approach. The review findings suggest that these approaches to qualitative methods need critical considerations, particularly when dealing with consumers of mental health services.

The review findings identified several sampling techniques used in mental health research. Quantitative studies, usually employ probability sampling, whilst qualitative studies use non-probability sampling [ 25 , 34 ]. The most common sampling techniques for quantitative studies are multi-stage sampling, which involves systematic, stratified, random sampling and consecutive sampling. In contrast, the predominant sampling approaches for qualitative studies are purposive [ 19 , 28 , 32 , 42 , 46 ], snowballing [ 30 , 32 , 46 ] and theoretical sampling [ 31 , 32 ].

The sampling of consumers of mental health services requires some important considerations. The sampling should consider the appropriateness and adequacy of the sampling approach, diversity and variety of consumers of services, attributes such as social, or cultural experiences, shared concerns related to the study, practical and organisational skills, as well as ethical and sensitivity issues are all relevant [ 31 , 32 , 46 ]. Sampling consumers of mental health services should also consider the homogeneity and heterogeneity of consumers. However, failure to address these considerations can present difficulty in sampling and subsequently result in selection and reporting bias in mental health research.

The evidence recommends several data collection approaches in collecting data in mental health research, including focus group discussion, extended in-depth interviews, observations, field notes, Delphi data collection and quasi-statistical techniques. The focus group discussions appear as an approach widely used to collect data from consumers of mental health services [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ]. The focus group discussion appears to be a significant source of obtaining information. This approach promotes the participation of consumers with severe conditions, particularly at the group level interaction. Mental health researchers are encouraged to use this approach to collect data from consumers, in order to promote group level interaction. Additionally, field notes can be used to supplement information and to more deeply analyse the interactions of consumers of mental health services. Field notes are significant when wanting to gather detailed accounts about the subjective experiences of consumers of mental health services [ 40 ]. Field notes can help researchers to capture the gestures and opinions of consumers of mental health services which cannot be covered in the audio-tape recording. Particularly, the field note is relevant to complement the richness of information collected through focus group discussion from consumers of mental health services.

Furthermore, it was found that in-depth interviews can be used to explore specific mental health issues, particularly culturally marked disorders, their personal and interpersonal significance, patient and family explanatory models, individual and family coping styles, as well as symptom symbols and protective mediators [ 19 , 30 , 34 ]. The in-depth interviews are particularly relevant if the study is interested in the lived experiences of consumers without the contamination of others in a group situation. The in-depth interviews are relevant when consumers of mental health services are uncomfortable in disclosing their confidential information in front of others [ 31 ]. The lived experience in a phenomenological context preferably allows the consumer the opportunity to express themselves anonymously without any tacit coercion created by a group context.

The review findings recommend significant factors requiring consideration when collecting data in mental health research. These considerations include building confidence and trust between the researcher and consumers [ 31 , 37 ], gaining confidence of mental health professionals who manage consumers of mental health services, seeking ethical approval from the relevant committees, meeting consumers of services before data collection as well as arranging a mutually acceptable venue for the groups and providing transport services [ 41 , 47 ]. The evidence confirms that the identification of consumers of mental health services to participate in research can be facilitated by mental health professionals. Similarly, the cultural and social differences of the consumers of mental health services need consideration when collecting data from them [ 26 , 31 ].

Moreover, our review advocates that standardised assessment instruments can be used to collect data from consumers of mental health services, particularly in quantitative data. The self-completion instruments for collecting such information can be converted to interviewer-completion instruments [ 33 , 49 ]. The interviewer can read the questions to consumers of mental health services and record their responses. It is recommended that collecting data from consumers of mental health services requires significant preparation, such as training with co-investigators and representatives from consumers of mental health services [ 25 , 30 , 33 , 34 , 35 , 39 , 49 ]. The training helps interviewers and other investigators to understand the research project, particularly translating and adapting an instrument for the study setting with the aim to achieve content, semantic, concept, criteria and technical equivalence [ 30 ]. The evidence indicates that there is a need to adequately train interviewers when preparing for fieldwork to collect data from consumers of mental health services.

The evidence provides several approaches that can be employed to ensure quality assurance in mental health research involving quantitative methods. The quality assurance approach encompasses seeking informed consent from consumers of mental health services [ 26 , 37 ], pre-testing of tools [ 25 , 49 ], minimising non-response rates and monitoring of the data collection process [ 25 , 33 , 49 ]. The quality assurance process in mental health research primarily aims to achieve the highest reliability, validity and timeliness, to improve the quality of care provided. For instance, the informed consent exposes consumers of mental health services to the aim(s), methods, anticipated benefits and potential hazards and discomforts of participating in the study. Herein, consumers of mental health services who cannot respond to the inform consent process because of the severity of their illness can have it signed by their family caregivers. The implication is that researchers should determine which category of consumers of mental health services need family caregivers involved in the consent process [ 37 ].

The review findings advises that researchers should use pre-testing to evaluate the data collection procedure on a small scale and then to subsequently make any necessary changes [ 25 ]. The pre-testing aims to help the interviewers get acquainted with the procedures and to detect any potential problems [ 49 ]. The researchers can discuss the findings of the pre-testing and then further resolve any challenges that may arise prior to the actual field work being commenced. The non-response rates in mental health research can be minimised by re-approaching consumers of mental health services who initially refuse to participate in the study.

In addition, quality assurance for qualitative data can be ensured by applying the principles of credibility, dependability, transferability, reflexivity, confirmability [ 19 , 32 ]. It was found that the credibility of qualitative research in mental health is achieved through methodological and interpretive rigour [ 32 , 37 ]. The methodological rigour for assessing credibility relates to congruence, responsiveness or sensitivity to a social context, appropriateness, adequacy and transparency. By contrast, ensuring interpretive rigour is achieved through authenticity, coherence, reciprocity, typicality and permeability of researchers’ intentions, engagement and interpretation [ 32 , 46 ].

Strengths and limitations

The evidence has several strengths and limitations that require interpretation and explanation. Firstly, we employed a systematic approach involving five stages of problem identification, literature search, data evaluation, data synthesis and presentation of results [ 21 ]. Similarly, we searched six databases and developed a data extraction form to extract information. The rigorous process employed in this study, for instance, searching databases and data extraction forms, helped to capture comprehensive information on the subject.

The integrative review has several limitations largely related to the search words, language limitations, time period and appraisal of methodological quality of included papers. In particular, the differences in key terms and words concerning methodological issues in the context of mental health research across cultures and organisational contexts may possibly have missed some relevant articles pertaining to the study. Similarly, limiting included studies to only English language articles and those published from January 2000 to July 2018 could have missed useful articles published in other languages and those published prior to 2000. The review did not assess the methodological quality of included papers using a critical appraisal tool, however, the combination of clearly articulated search methods, consultation with the research librarian, and reviewing articles with methodological experts in mental health research helped to address the limitations.

The review identified several methodological issues that need critical attention when conducting mental health research. The evidence confirms that studies that addressed methodological considerations in conducting mental health research largely focuses on qualitative studies in a transcultural setting, in addition to lessons from multi-site surveys in mental health research. Specifically, the methodological issues related to the study design, sampling, data collection processes and quality assurance are critical to the research design chosen for any particular study. The review highlighted that researchers conducting mental health research can establish early collaboration, familiarise themselves with the topic, share information on the topic, negotiate to resolve any emerging problems and seek the contribution of clinical (or researcher) team members on the ground. In addition, the recruitment of consumers of mental health services should consider the appropriateness and adequacy of sampling approaches, diversity and variety of consumers of services, their social or cultural experiences, practical and organisational skills, as well as ethical and sensitivity issues.

The evidence confirms that in an attempt to effectively recruit and collect data from consumers of mental health services, there is the need to build confidence and trust between the researcher and consumers; and to gain the confidence of mental health service providers. Furthermore, seeking ethical approval from the relevant committee, meeting with consumers of services before data collection, arranging a mutually acceptable venue for the groups, and providing transport services, are all further important considerations. The review findings establish that researchers conducting mental health research should consider several quality assurance issues. Issues such as adequate training prior to data collection, seeking informed consent from consumers of mental health services, pre-testing of tools, minimising non-response rates and monitoring of the data collection process. More specifically, quality assurance for qualitative data can be achieved by applying the principles of credibility, dependability, transferability, reflexivity, confirmability.

Based on the findings from this review, it is recommended that mental health research should adequately consider the methodological issues regarding study design, sampling, data collection procedures and quality assurance issues to effectively conduct meaningful research.

Availability of data and materials

Not applicable

Abbreviations

focus group discussions

Joanna Briggs Institute

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

National Ethics Advisory Committee. Ethical guidelines for intervention studies: revised edition. Wellington (New Zealand): Ministry of Health. 2012.

Mann C. Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J. 2003;20(1):54–60.

Article   CAS   Google Scholar  

DiPietro NA. Methods in epidemiology: observational study designs. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy. 2010;30(10):973–84.

Article   Google Scholar  

Hong NQ, Pluyr P, Fabregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed Methods Appraisal Tool (MMAT). Canada.: Intellectual Property Office, Canada; 2018.

Creswell JW, Creswell JD. Research design: qualitative, quantitative, and mixed methods approaches: sage publications; 2017.

Google Scholar  

Wisdom J, Creswell JW. Mixed methods: integrating quantitative and qualitative data collection and analysis while studying patient-centered medical home models. Rockville: Agency for Healthcare Research and Quality; 2013.

Bonita R, Beaglehole R, Kjellström T. Basic epidemiology: World Health Organization; 2006.

Centers for Disease Control Prevention [CDC]. Principles of epidemiology in public health practice: an introduction to applied epidemiology and biostatistics. Atlanta, GA: US Dept. of Health and Human Services, Centers for Disease Control and Prevention (CDC), Office of Workforce and Career Development; 2012.

Parab S, Bhalerao S. Study designs. International journal of Ayurveda research. 2010;1(2):128.

Yang W, Zilov A, Soewondo P, Bech OM, Sekkal F, Home PD. Observational studies: going beyond the boundaries of randomized controlled trials. Diabetes Res Clin Pract. 2010;88:S3–9.

Department of Family Medicine (McGill University). Mixed Methods Appraisal Tool (MMAT) – Version 2011 Canada: McGill University; 2011 [Available from: http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/84371689/MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.08.21.pdf .

Besen J, Gan DS. A critical evaluation of clinical research study designs. J Investig Dermatol. 2014;134.

Axelrod DA, Hayward R. Nonrandomized interventional study designs (quasi-experimental designs). Clinical research methods for surgeons: Springer; 2006. p. 63–76.

Thiese MS. Observational and interventional study design types; an overview. Biochemia medica: Biochemia medica. 2014;24(2):199–210.

Velengtas P, Mohr P, Messner DA. Making informed decisions: assessing the strengths and weaknesses of study designs and analytic methods for comparative effectiveness research. National Pharmaceutical Council 2012.

Guerrera F, Renaud S, Tabbò F, Filosso PL. How to design a randomized clinical trial: tips and tricks for conduct a successful study in thoracic disease domain. Journal of thoracic disease. 2017;9(8):2692.

Bhide A, Shah PS, Acharya G. A simplified guide to randomized controlled trials. Acta Obstet Gynecol Scand. 2018;97(4):380–7.

Palinkas L, Horwitz SM, Chamberlain P, Hurlburt MS, Landsverk J. Mixed-methods designs in mental health services research: a review. Psychiatr Serv. 2011;62(3):255–63.

Palinkas L. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychol. 2014;43(6):851–61.

World Health Organization [WHO]. Mental health: a state of well-being 2014 [Available from: http://www.who.int/features/factfiles/mental_health/en/ .

Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546–53.

Hopia H, Latvala E, Liimatainen L. Reviewing the methodology of an integrative review. Scand J Caring Sci. 2016;30(4):662–9.

Pearson A, White H, Bath-Hextall F, Apostolo J, Salmond S, Kirkpatrick P. Methodology for JBI mixed methods systematic reviews. The Joanna Briggs Institute Reviewers Manual. 2014;1:5–34.

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Alonso J, Angermeyer MC, Bernert S, Bruffaerts R, Brugha TS, Bryson H, et al. Sampling and methods of the European study of the epidemiology of mental disorders (ESEMeD) project. Acta Psychiatr Scand Suppl. 2004;109(420):8–20.

Baarnhielm S, Ekblad S. Qualitative research, culture and ethics: a case discussion. Transcultural Psychiatry. 2002;39(4):469–83.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Brown C, Lloyd K. Qualitative methods in psychiatric research. Adv Psychiatr Treat. 2001;7(5):350–6.

Davidsen AS. Phenomenological approaches in psychology and health sciences. Qual Res Psychol. 2013;10(3):318–39.

de Jong JT, Van Ommeren M. Toward a culture-informed epidemiology: combining qualitative and quantitative research in transcultural contexts. Transcultural Psychiatry. 2002;39(4):422–33.

Ekblad S, Baarnhielm S. Focus group interview research in transcultural psychiatry: reflections on research experiences. Transcultural Psychiatry. 2002;39(4):484–500.

Fossey E, Harvey C, McDermott F, Davidson L. Understanding and evaluating qualitative research. Aust N Z J Psychiatry. 2002;36(6):717–32.

Jacobi F, Wittchen H-U, Holting C, Sommer S, Lieb R, Hofler M, et al. Estimating the prevalence of mental and somatic disorders in the community: aims and methods of the German National Health Interview and examination survey. Int J Methods Psychiatr Res 2002;11(1):1–18.

Koch A, Vogel A, Holzmann M, Pfennig A, Salize HJ, Puschner B, et al. MEMENTA-‘Mental healthcare provision for adults with intellectual disability and a mental disorder’. A cross-sectional epidemiological multisite study assessing prevalence of psychiatric symptomatology, needs for care and quality of healthcare provision for adults with intellectual disability in Germany: a study protocol. BMJ Open. 2014;4(5):e004878.

Korver N, Quee PJ, Boos HB, Simons CJ, de Haan L, Investigators G. Genetic risk and outcome of psychosis (GROUP), a multi site longitudinal cohort study focused on gene–environment interaction: objectives, sample characteristics, recruitment and assessment methods. Int J Methods Psychiatr Res. 2012;21(3):205–21.

Larkin M, Watts S, Clifton E. Giving voice and making sense in interpretative phenomenological analysis. Qual Res Psychol. 2006;3(2):102–20.

Latvala E, Vuokila-Oikkonen P, Janhonen S. Videotaped recording as a method of participant observation in psychiatric nursing research. J Adv Nurs. 2000;31(5):1252–7.

Leese MN, White IR, Schene AH, Koeter MW, Ruggeri M, Gaite L. Reliability in multi-site psychiatric studies. Int J Methods Psychiatr Res. 2001;10(1):29–42.

Liu Z, Huang Y, Lv P, Zhang T, Wang H, Li Q, et al. The China mental health survey: II. Design and field procedures. Soc Psychiatry Psychiatr Epidemiol. 2016;51(11):1547–57.

Montgomery P, Bailey PH. Field notes and theoretical memos in grounded theory. West J Nurs Res. 2007;29(1):65–79.

Owen S. The practical, methodological and ethical dilemmas of conducting focus groups with vulnerable clients. J Adv Nurs. 2001;36(5):652–8.

Palinkas L, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health Ment Health Serv Res. 2015;42(5):533–44.

Palinkas L, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):44–53.

Razafsha M, Behforuzi H, Azari H, Zhang Z, Wang KK, Kobeissy FH, et al. Qualitative versus quantitative methods in psychiatric research. Methods Mol Biol. 2012;829:49–62.

Robins CS, Ware NC, Dosreis S, Willging CE, Chung JY, Lewis-Fernández R. Dialogues on mixed-methods and mental health services research: anticipating challenges, building solutions. Psychiatr Serv. 2008;59(7):727–31.

Robinson OC. Sampling in interview-based qualitative research: a theoretical and practical guide. Qual Res Psychol. 2014;11(1):25–41.

Schilder K, Tomov T, Mladenova M, Mayeya J, Jenkins R, Gulbinat W, et al. The appropriateness and use of focus group methodology across international mental health communities. International Review of Psychiatry. 2004;16(1–2):24–30.

Schoonenboom J, Johnson RB. How to construct a mixed methods research DesignWie man ein mixed methods-Forschungs-design konstruiert. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie. 2017;69(2):107–31.

Yin H, Phillips MR, Wardenaar KJ, Xu G, Ormel J, Tian H, et al. The Tianjin mental health survey (TJMHS): study rationale, design and methods. Int J Methods Psychiatr Res. 2017;26(3):09.

Download references

Acknowledgements

The authors wish to thank the University of Newcastle Graduate Research and the School of Nursing and Midwifery, for the Doctoral Scholarship offered to the lead author. The authors are also grateful for the support received from Ms. Debbie Booth, the Librarian for supporting the literature search.

Author information

Authors and affiliations.

School of Nursing and Midwifery, The University of Newcastle, Callaghan, Australia

Faculty of Health and Medicine, School Nursing and Midwifery, University of Newcastle, Callaghan, Australia

Anthony Paul O’Brien

Faculty of Business and Economics, Macquarie University, North Ryde, Australia

Rebecca Mitchell

You can also search for this author in PubMed   Google Scholar

Contributions

EB, APO’B, and RM conceptualized the study. EB conducted the data extraction, APO’B, and RM, conducted the second review of the extracted data. EB, working closely with APO’B and RM performed the content analysis and drafted the manuscript. EB, APO’B, and RM, reviewed and made inputs into the intellectual content and agreed on its submission for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Eric Badu .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Data extraction form. (DOCX 18 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Badu, E., O’Brien, A.P. & Mitchell, R. An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance. Arch Public Health 77 , 37 (2019). https://doi.org/10.1186/s13690-019-0363-z

Download citation

Received : 13 November 2018

Accepted : 22 July 2019

Published : 10 October 2019

DOI : https://doi.org/10.1186/s13690-019-0363-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mental health
• Methodological approach
• Mixed methods
• Data collection

Archives of Public Health

ISSN: 2049-3258

• General enquiries: [email protected]

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Recent quantitative research on determinants of health in high income countries: A scoping review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Centre for Health Economics Research and Modelling Infectious Diseases, Vaccine and Infectious Disease Institute, University of Antwerp, Antwerp, Belgium

Roles Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

• Vladimira Varbanova, 
• Philippe Beutels

• Published: September 17, 2020
• https://doi.org/10.1371/journal.pone.0239031
• Peer Review
• Reader Comments

Identifying determinants of health and understanding their role in health production constitutes an important research theme. We aimed to document the state of recent multi-country research on this theme in the literature.

We followed the PRISMA-ScR guidelines to systematically identify, triage and review literature (January 2013—July 2019). We searched for studies that performed cross-national statistical analyses aiming to evaluate the impact of one or more aggregate level determinants on one or more general population health outcomes in high-income countries. To assess in which combinations and to what extent individual (or thematically linked) determinants had been studied together, we performed multidimensional scaling and cluster analysis.

Sixty studies were selected, out of an original yield of 3686. Life-expectancy and overall mortality were the most widely used population health indicators, while determinants came from the areas of healthcare, culture, politics, socio-economics, environment, labor, fertility, demographics, life-style, and psychology. The family of regression models was the predominant statistical approach. Results from our multidimensional scaling showed that a relatively tight core of determinants have received much attention, as main covariates of interest or controls, whereas the majority of other determinants were studied in very limited contexts. We consider findings from these studies regarding the importance of any given health determinant inconclusive at present. Across a multitude of model specifications, different country samples, and varying time periods, effects fluctuated between statistically significant and not significant, and between beneficial and detrimental to health.

Conclusions

We conclude that efforts to understand the underlying mechanisms of population health are far from settled, and the present state of research on the topic leaves much to be desired. It is essential that future research considers multiple factors simultaneously and takes advantage of more sophisticated methodology with regards to quantifying health as well as analyzing determinants’ influence.

Citation: Varbanova V, Beutels P (2020) Recent quantitative research on determinants of health in high income countries: A scoping review. PLoS ONE 15(9): e0239031. https://doi.org/10.1371/journal.pone.0239031

Editor: Amir Radfar, University of Central Florida, UNITED STATES

Received: November 14, 2019; Accepted: August 28, 2020; Published: September 17, 2020

Copyright: © 2020 Varbanova, Beutels. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: This study (and VV) is funded by the Research Foundation Flanders ( https://www.fwo.be/en/ ), FWO project number G0D5917N, award obtained by PB. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Identifying the key drivers of population health is a core subject in public health and health economics research. Between-country comparative research on the topic is challenging. In order to be relevant for policy, it requires disentangling different interrelated drivers of “good health”, each having different degrees of importance in different contexts.

“Good health”–physical and psychological, subjective and objective–can be defined and measured using a variety of approaches, depending on which aspect of health is the focus. A major distinction can be made between health measurements at the individual level or some aggregate level, such as a neighborhood, a region or a country. In view of this, a great diversity of specific research topics exists on the drivers of what constitutes individual or aggregate “good health”, including those focusing on health inequalities, the gender gap in longevity, and regional mortality and longevity differences.

The current scoping review focuses on determinants of population health. Stated as such, this topic is quite broad. Indeed, we are interested in the very general question of what methods have been used to make the most of increasingly available region or country-specific databases to understand the drivers of population health through inter-country comparisons. Existing reviews indicate that researchers thus far tend to adopt a narrower focus. Usually, attention is given to only one health outcome at a time, with further geographical and/or population [ 1 , 2 ] restrictions. In some cases, the impact of one or more interventions is at the core of the review [ 3 – 7 ], while in others it is the relationship between health and just one particular predictor, e.g., income inequality, access to healthcare, government mechanisms [ 8 – 13 ]. Some relatively recent reviews on the subject of social determinants of health [ 4 – 6 , 14 – 17 ] have considered a number of indicators potentially influencing health as opposed to a single one. One review defines “social determinants” as “the social, economic, and political conditions that influence the health of individuals and populations” [ 17 ] while another refers even more broadly to “the factors apart from medical care” [ 15 ].

In the present work, we aimed to be more inclusive, setting no limitations on the nature of possible health correlates, as well as making use of a multitude of commonly accepted measures of general population health. The goal of this scoping review was to document the state of the art in the recent published literature on determinants of population health, with a particular focus on the types of determinants selected and the methodology used. In doing so, we also report the main characteristics of the results these studies found. The materials collected in this review are intended to inform our (and potentially other researchers’) future analyses on this topic. Since the production of health is subject to the law of diminishing marginal returns, we focused our review on those studies that included countries where a high standard of wealth has been achieved for some time, i.e., high-income countries belonging to the Organisation for Economic Co-operation and Development (OECD) or Europe. Adding similar reviews for other country income groups is of limited interest to the research we plan to do in this area.

In view of its focus on data and methods, rather than results, a formal protocol was not registered prior to undertaking this review, but the procedure followed the guidelines of the PRISMA statement for scoping reviews [ 18 ].

We focused on multi-country studies investigating the potential associations between any aggregate level (region/city/country) determinant and general measures of population health (e.g., life expectancy, mortality rate).

Within the query itself, we listed well-established population health indicators as well as the six world regions, as defined by the World Health Organization (WHO). We searched only in the publications’ titles in order to keep the number of hits manageable, and the ratio of broadly relevant abstracts over all abstracts in the order of magnitude of 10% (based on a series of time-focused trial runs). The search strategy was developed iteratively between the two authors and is presented in S1 Appendix . The search was performed by VV in PubMed and Web of Science on the 16 th of July, 2019, without any language restrictions, and with a start date set to the 1 st of January, 2013, as we were interested in the latest developments in this area of research.

Eligibility criteria

Records obtained via the search methods described above were screened independently by the two authors. Consistency between inclusion/exclusion decisions was approximately 90% and the 43 instances where uncertainty existed were judged through discussion. Articles were included subject to meeting the following requirements: (a) the paper was a full published report of an original empirical study investigating the impact of at least one aggregate level (city/region/country) factor on at least one health indicator (or self-reported health) of the general population (the only admissible “sub-populations” were those based on gender and/or age); (b) the study employed statistical techniques (calculating correlations, at the very least) and was not purely descriptive or theoretical in nature; (c) the analysis involved at least two countries or at least two regions or cities (or another aggregate level) in at least two different countries; (d) the health outcome was not differentiated according to some socio-economic factor and thus studied in terms of inequality (with the exception of gender and age differentiations); (e) mortality, in case it was one of the health indicators under investigation, was strictly “total” or “all-cause” (no cause-specific or determinant-attributable mortality).

Data extraction

The following pieces of information were extracted in an Excel table from the full text of each eligible study (primarily by VV, consulting with PB in case of doubt): health outcome(s), determinants, statistical methodology, level of analysis, results, type of data, data sources, time period, countries. The evidence is synthesized according to these extracted data (often directly reflected in the section headings), using a narrative form accompanied by a “summary-of-findings” table and a graph.

Search and selection

The initial yield contained 4583 records, reduced to 3686 after removal of duplicates ( Fig 1 ). Based on title and abstract screening, 3271 records were excluded because they focused on specific medical condition(s) or specific populations (based on morbidity or some other factor), dealt with intervention effectiveness, with theoretical or non-health related issues, or with animals or plants. Of the remaining 415 papers, roughly half were disqualified upon full-text consideration, mostly due to using an outcome not of interest to us (e.g., health inequality), measuring and analyzing determinants and outcomes exclusively at the individual level, performing analyses one country at a time, employing indices that are a mixture of both health indicators and health determinants, or not utilizing potential health determinants at all. After this second stage of the screening process, 202 papers were deemed eligible for inclusion. This group was further dichotomized according to level of economic development of the countries or regions under study, using membership of the OECD or Europe as a reference “cut-off” point. Sixty papers were judged to include high-income countries, and the remaining 142 included either low- or middle-income countries or a mix of both these levels of development. The rest of this report outlines findings in relation to high-income countries only, reflecting our own primary research interests. Nonetheless, we chose to report our search yield for the other income groups for two reasons. First, to gauge the relative interest in applied published research for these different income levels; and second, to enable other researchers with a focus on determinants of health in other countries to use the extraction we made here.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0239031.g001

Health outcomes

The most frequent population health indicator, life expectancy (LE), was present in 24 of the 60 studies. Apart from “life expectancy at birth” (representing the average life-span a newborn is expected to have if current mortality rates remain constant), also called “period LE” by some [ 19 , 20 ], we encountered as well LE at 40 years of age [ 21 ], at 60 [ 22 ], and at 65 [ 21 , 23 , 24 ]. In two papers, the age-specificity of life expectancy (be it at birth or another age) was not stated [ 25 , 26 ].

Some studies considered male and female LE separately [ 21 , 24 , 25 , 27 – 33 ]. This consideration was also often observed with the second most commonly used health index [ 28 – 30 , 34 – 38 ]–termed “total”, or “overall”, or “all-cause”, mortality rate (MR)–included in 22 of the 60 studies. In addition to gender, this index was also sometimes broken down according to age group [ 30 , 39 , 40 ], as well as gender-age group [ 38 ].

While the majority of studies under review here focused on a single health indicator, 23 out of the 60 studies made use of multiple outcomes, although these outcomes were always considered one at a time, and sometimes not all of them fell within the scope of our review. An easily discernable group of indices that typically went together [ 25 , 37 , 41 ] was that of neonatal (deaths occurring within 28 days postpartum), perinatal (fetal or early neonatal / first-7-days deaths), and post-neonatal (deaths between the 29 th day and completion of one year of life) mortality. More often than not, these indices were also accompanied by “stand-alone” indicators, such as infant mortality (deaths within the first year of life; our third most common index found in 16 of the 60 studies), maternal mortality (deaths during pregnancy or within 42 days of termination of pregnancy), and child mortality rates. Child mortality has conventionally been defined as mortality within the first 5 years of life, thus often also called “under-5 mortality”. Nonetheless, Pritchard & Wallace used the term “child mortality” to denote deaths of children younger than 14 years [ 42 ].

As previously stated, inclusion criteria did allow for self-reported health status to be used as a general measure of population health. Within our final selection of studies, seven utilized some form of subjective health as an outcome variable [ 25 , 43 – 48 ]. Additionally, the Health Human Development Index [ 49 ], healthy life expectancy [ 50 ], old-age survival [ 51 ], potential years of life lost [ 52 ], and disability-adjusted life expectancy [ 25 ] were also used.

We note that while in most cases the indicators mentioned above (and/or the covariates considered, see below) were taken in their absolute or logarithmic form, as a—typically annual—number, sometimes they were used in the form of differences, change rates, averages over a given time period, or even z-scores of rankings [ 19 , 22 , 40 , 42 , 44 , 53 – 57 ].

Regions, countries, and populations

Despite our decision to confine this review to high-income countries, some variation in the countries and regions studied was still present. Selection seemed to be most often conditioned on the European Union, or the European continent more generally, and the Organisation of Economic Co-operation and Development (OECD), though, typically, not all member nations–based on the instances where these were also explicitly listed—were included in a given study. Some of the stated reasons for omitting certain nations included data unavailability [ 30 , 45 , 54 ] or inconsistency [ 20 , 58 ], Gross Domestic Product (GDP) too low [ 40 ], differences in economic development and political stability with the rest of the sampled countries [ 59 ], and national population too small [ 24 , 40 ]. On the other hand, the rationales for selecting a group of countries included having similar above-average infant mortality [ 60 ], similar healthcare systems [ 23 ], and being randomly drawn from a social spending category [ 61 ]. Some researchers were interested explicitly in a specific geographical region, such as Eastern Europe [ 50 ], Central and Eastern Europe [ 48 , 60 ], the Visegrad (V4) group [ 62 ], or the Asia/Pacific area [ 32 ]. In certain instances, national regions or cities, rather than countries, constituted the units of investigation instead [ 31 , 51 , 56 , 62 – 66 ]. In two particular cases, a mix of countries and cities was used [ 35 , 57 ]. In another two [ 28 , 29 ], due to the long time periods under study, some of the included countries no longer exist. Finally, besides “European” and “OECD”, the terms “developed”, “Western”, and “industrialized” were also used to describe the group of selected nations [ 30 , 42 , 52 , 53 , 67 ].

As stated above, it was the health status of the general population that we were interested in, and during screening we made a concerted effort to exclude research using data based on a more narrowly defined group of individuals. All studies included in this review adhere to this general rule, albeit with two caveats. First, as cities (even neighborhoods) were the unit of analysis in three of the studies that made the selection [ 56 , 64 , 65 ], the populations under investigation there can be more accurately described as general urban , instead of just general. Second, oftentimes health indicators were stratified based on gender and/or age, therefore we also admitted one study that, due to its specific research question, focused on men and women of early retirement age [ 35 ] and another that considered adult males only [ 68 ].

Data types and sources

A great diversity of sources was utilized for data collection purposes. The accessible reference databases of the OECD ( https://www.oecd.org/ ), WHO ( https://www.who.int/ ), World Bank ( https://www.worldbank.org/ ), United Nations ( https://www.un.org/en/ ), and Eurostat ( https://ec.europa.eu/eurostat ) were among the top choices. The other international databases included Human Mortality [ 30 , 39 , 50 ], Transparency International [ 40 , 48 , 50 ], Quality of Government [ 28 , 69 ], World Income Inequality [ 30 ], International Labor Organization [ 41 ], International Monetary Fund [ 70 ]. A number of national databases were referred to as well, for example the US Bureau of Statistics [ 42 , 53 ], Korean Statistical Information Services [ 67 ], Statistics Canada [ 67 ], Australian Bureau of Statistics [ 67 ], and Health New Zealand Tobacco control and Health New Zealand Food and Nutrition [ 19 ]. Well-known surveys, such as the World Values Survey [ 25 , 55 ], the European Social Survey [ 25 , 39 , 44 ], the Eurobarometer [ 46 , 56 ], the European Value Survey [ 25 ], and the European Statistics of Income and Living Condition Survey [ 43 , 47 , 70 ] were used as data sources, too. Finally, in some cases [ 25 , 28 , 29 , 35 , 36 , 41 , 69 ], built-for-purpose datasets from previous studies were re-used.

In most of the studies, the level of the data (and analysis) was national. The exceptions were six papers that dealt with Nomenclature of Territorial Units of Statistics (NUTS2) regions [ 31 , 62 , 63 , 66 ], otherwise defined areas [ 51 ] or cities [ 56 ], and seven others that were multilevel designs and utilized both country- and region-level data [ 57 ], individual- and city- or country-level [ 35 ], individual- and country-level [ 44 , 45 , 48 ], individual- and neighborhood-level [ 64 ], and city-region- (NUTS3) and country-level data [ 65 ]. Parallel to that, the data type was predominantly longitudinal, with only a few studies using purely cross-sectional data [ 25 , 33 , 43 , 45 – 48 , 50 , 62 , 67 , 68 , 71 , 72 ], albeit in four of those [ 43 , 48 , 68 , 72 ] two separate points in time were taken (thus resulting in a kind of “double cross-section”), while in another the averages across survey waves were used [ 56 ].

In studies using longitudinal data, the length of the covered time periods varied greatly. Although this was almost always less than 40 years, in one study it covered the entire 20 th century [ 29 ]. Longitudinal data, typically in the form of annual records, was sometimes transformed before usage. For example, some researchers considered data points at 5- [ 34 , 36 , 49 ] or 10-year [ 27 , 29 , 35 ] intervals instead of the traditional 1, or took averages over 3-year periods [ 42 , 53 , 73 ]. In one study concerned with the effect of the Great Recession all data were in a “recession minus expansion change in trends”-form [ 57 ]. Furthermore, there were a few instances where two different time periods were compared to each other [ 42 , 53 ] or when data was divided into 2 to 4 (possibly overlapping) periods which were then analyzed separately [ 24 , 26 , 28 , 29 , 31 , 65 ]. Lastly, owing to data availability issues, discrepancies between the time points or periods of data on the different variables were occasionally observed [ 22 , 35 , 42 , 53 – 55 , 63 ].

Health determinants

Together with other essential details, Table 1 lists the health correlates considered in the selected studies. Several general categories for these correlates can be discerned, including health care, political stability, socio-economics, demographics, psychology, environment, fertility, life-style, culture, labor. All of these, directly or implicitly, have been recognized as holding importance for population health by existing theoretical models of (social) determinants of health [ 74 – 77 ].

https://doi.org/10.1371/journal.pone.0239031.t001

It is worth noting that in a few studies there was just a single aggregate-level covariate investigated in relation to a health outcome of interest to us. In one instance, this was life satisfaction [ 44 ], in another–welfare system typology [ 45 ], but also gender inequality [ 33 ], austerity level [ 70 , 78 ], and deprivation [ 51 ]. Most often though, attention went exclusively to GDP [ 27 , 29 , 46 , 57 , 65 , 71 ]. It was often the case that research had a more particular focus. Among others, minimum wages [ 79 ], hospital payment schemes [ 23 ], cigarette prices [ 63 ], social expenditure [ 20 ], residents’ dissatisfaction [ 56 ], income inequality [ 30 , 69 ], and work leave [ 41 , 58 ] took center stage. Whenever variables outside of these specific areas were also included, they were usually identified as confounders or controls, moderators or mediators.

We visualized the combinations in which the different determinants have been studied in Fig 2 , which was obtained via multidimensional scaling and a subsequent cluster analysis (details outlined in S2 Appendix ). It depicts the spatial positioning of each determinant relative to all others, based on the number of times the effects of each pair of determinants have been studied simultaneously. When interpreting Fig 2 , one should keep in mind that determinants marked with an asterisk represent, in fact, collectives of variables.

Groups of determinants are marked by asterisks (see S1 Table in S1 Appendix ). Diminishing color intensity reflects a decrease in the total number of “connections” for a given determinant. Noteworthy pairwise “connections” are emphasized via lines (solid-dashed-dotted indicates decreasing frequency). Grey contour lines encircle groups of variables that were identified via cluster analysis. Abbreviations: age = population age distribution, associations = membership in associations, AT-index = atherogenic-thrombogenic index, BR = birth rate, CAPB = Cyclically Adjusted Primary Balance, civilian-labor = civilian labor force, C-section = Cesarean delivery rate, credit-info = depth of credit information, dissatisf = residents’ dissatisfaction, distrib.orient = distributional orientation, EDU = education, eHealth = eHealth index at GP-level, exch.rate = exchange rate, fat = fat consumption, GDP = gross domestic product, GFCF = Gross Fixed Capital Formation/Creation, GH-gas = greenhouse gas, GII = gender inequality index, gov = governance index, gov.revenue = government revenues, HC-coverage = healthcare coverage, HE = health(care) expenditure, HHconsump = household consumption, hosp.beds = hospital beds, hosp.payment = hospital payment scheme, hosp.stay = length of hospital stay, IDI = ICT development index, inc.ineq = income inequality, industry-labor = industrial labor force, infant-sex = infant sex ratio, labor-product = labor production, LBW = low birth weight, leave = work leave, life-satisf = life satisfaction, M-age = maternal age, marginal-tax = marginal tax rate, MDs = physicians, mult.preg = multiple pregnancy, NHS = Nation Health System, NO = nitrous oxide emissions, PM10 = particulate matter (PM10) emissions, pop = population size, pop.density = population density, pre-term = pre-term birth rate, prison = prison population, researchE = research&development expenditure, school.ref = compulsory schooling reform, smoke-free = smoke-free places, SO = sulfur oxide emissions, soc.E = social expenditure, soc.workers = social workers, sugar = sugar consumption, terror = terrorism, union = union density, UR = unemployment rate, urban = urbanization, veg-fr = vegetable-and-fruit consumption, welfare = welfare regime, Wwater = wastewater treatment.

https://doi.org/10.1371/journal.pone.0239031.g002

Distances between determinants in Fig 2 are indicative of determinants’ “connectedness” with each other. While the statistical procedure called for higher dimensionality of the model, for demonstration purposes we show here a two-dimensional solution. This simplification unfortunately comes with a caveat. To use the factor smoking as an example, it would appear it stands at a much greater distance from GDP than it does from alcohol. In reality however, smoking was considered together with alcohol consumption [ 21 , 25 , 26 , 52 , 68 ] in just as many studies as it was with GDP [ 21 , 25 , 26 , 52 , 59 ], five. To aid with respect to this apparent shortcoming, we have emphasized the strongest pairwise links. Solid lines connect GDP with health expenditure (HE), unemployment rate (UR), and education (EDU), indicating that the effect of GDP on health, taking into account the effects of the other three determinants as well, was evaluated in between 12 to 16 studies of the 60 included in this review. Tracing the dashed lines, we can also tell that GDP appeared jointly with income inequality, and HE together with either EDU or UR, in anywhere between 8 to 10 of our selected studies. Finally, some weaker but still worth-mentioning “connections” between variables are displayed as well via the dotted lines.

The fact that all notable pairwise “connections” are concentrated within a relatively small region of the plot may be interpreted as low overall “connectedness” among the health indicators studied. GDP is the most widely investigated determinant in relation to general population health. Its total number of “connections” is disproportionately high (159) compared to its runner-up–HE (with 113 “connections”), and then subsequently EDU (with 90) and UR (with 86). In fact, all of these determinants could be thought of as outliers, given that none of the remaining factors have a total count of pairings above 52. This decrease in individual determinants’ overall “connectedness” can be tracked on the graph via the change of color intensity as we move outwards from the symbolic center of GDP and its closest “co-determinants”, to finally reach the other extreme of the ten indicators (welfare regime, household consumption, compulsory school reform, life satisfaction, government revenues, literacy, research expenditure, multiple pregnancy, Cyclically Adjusted Primary Balance, and residents’ dissatisfaction; in white) the effects on health of which were only studied in isolation.

Lastly, we point to the few small but stable clusters of covariates encircled by the grey bubbles on Fig 2 . These groups of determinants were identified as “close” by both statistical procedures used for the production of the graph (see details in S2 Appendix ).

Statistical methodology

There was great variation in the level of statistical detail reported. Some authors provided too vague a description of their analytical approach, necessitating some inference in this section.

The issue of missing data is a challenging reality in this field of research, but few of the studies under review (12/60) explain how they dealt with it. Among the ones that do, three general approaches to handling missingness can be identified, listed in increasing level of sophistication: case-wise deletion, i.e., removal of countries from the sample [ 20 , 45 , 48 , 58 , 59 ], (linear) interpolation [ 28 , 30 , 34 , 58 , 59 , 63 ], and multiple imputation [ 26 , 41 , 52 ].

Correlations, Pearson, Spearman, or unspecified, were the only technique applied with respect to the health outcomes of interest in eight analyses [ 33 , 42 – 44 , 46 , 53 , 57 , 61 ]. Among the more advanced statistical methods, the family of regression models proved to be, by and large, predominant. Before examining this closer, we note the techniques that were, in a way, “unique” within this selection of studies: meta-analyses were performed (random and fixed effects, respectively) on the reduced form and 2-sample two stage least squares (2SLS) estimations done within countries [ 39 ]; difference-in-difference (DiD) analysis was applied in one case [ 23 ]; dynamic time-series methods, among which co-integration, impulse-response function (IRF), and panel vector autoregressive (VAR) modeling, were utilized in one study [ 80 ]; longitudinal generalized estimating equation (GEE) models were developed on two occasions [ 70 , 78 ]; hierarchical Bayesian spatial models [ 51 ] and special autoregressive regression [ 62 ] were also implemented.

Purely cross-sectional data analyses were performed in eight studies [ 25 , 45 , 47 , 50 , 55 , 56 , 67 , 71 ]. These consisted of linear regression (assumed ordinary least squares (OLS)), generalized least squares (GLS) regression, and multilevel analyses. However, six other studies that used longitudinal data in fact had a cross-sectional design, through which they applied regression at multiple time-points separately [ 27 , 29 , 36 , 48 , 68 , 72 ].

Apart from these “multi-point cross-sectional studies”, some other simplistic approaches to longitudinal data analysis were found, involving calculating and regressing 3-year averages of both the response and the predictor variables [ 54 ], taking the average of a few data-points (i.e., survey waves) [ 56 ] or using difference scores over 10-year [ 19 , 29 ] or unspecified time intervals [ 40 , 55 ].

Moving further in the direction of more sensible longitudinal data usage, we turn to the methods widely known among (health) economists as “panel data analysis” or “panel regression”. Most often seen were models with fixed effects for country/region and sometimes also time-point (occasionally including a country-specific trend as well), with robust standard errors for the parameter estimates to take into account correlations among clustered observations [ 20 , 21 , 24 , 28 , 30 , 32 , 34 , 37 , 38 , 41 , 52 , 59 , 60 , 63 , 66 , 69 , 73 , 79 , 81 , 82 ]. The Hausman test [ 83 ] was sometimes mentioned as the tool used to decide between fixed and random effects [ 26 , 49 , 63 , 66 , 73 , 82 ]. A few studies considered the latter more appropriate for their particular analyses, with some further specifying that (feasible) GLS estimation was employed [ 26 , 34 , 49 , 58 , 60 , 73 ]. Apart from these two types of models, the first differences method was encountered once as well [ 31 ]. Across all, the error terms were sometimes assumed to come from a first-order autoregressive process (AR(1)), i.e., they were allowed to be serially correlated [ 20 , 30 , 38 , 58 – 60 , 73 ], and lags of (typically) predictor variables were included in the model specification, too [ 20 , 21 , 37 , 38 , 48 , 69 , 81 ]. Lastly, a somewhat different approach to longitudinal data analysis was undertaken in four studies [ 22 , 35 , 48 , 65 ] in which multilevel–linear or Poisson–models were developed.

Regardless of the exact techniques used, most studies included in this review presented multiple model applications within their main analysis. None attempted to formally compare models in order to identify the “best”, even if goodness-of-fit statistics were occasionally reported. As indicated above, many studies investigated women’s and men’s health separately [ 19 , 21 , 22 , 27 – 29 , 31 , 33 , 35 , 36 , 38 , 39 , 45 , 50 , 51 , 64 , 65 , 69 , 82 ], and covariates were often tested one at a time, including other covariates only incrementally [ 20 , 25 , 28 , 36 , 40 , 50 , 55 , 67 , 73 ]. Furthermore, there were a few instances where analyses within countries were performed as well [ 32 , 39 , 51 ] or where the full time period of interest was divided into a few sub-periods [ 24 , 26 , 28 , 31 ]. There were also cases where different statistical techniques were applied in parallel [ 29 , 55 , 60 , 66 , 69 , 73 , 82 ], sometimes as a form of sensitivity analysis [ 24 , 26 , 30 , 58 , 73 ]. However, the most common approach to sensitivity analysis was to re-run models with somewhat different samples [ 39 , 50 , 59 , 67 , 69 , 80 , 82 ]. Other strategies included different categorization of variables or adding (more/other) controls [ 21 , 23 , 25 , 28 , 37 , 50 , 63 , 69 ], using an alternative main covariate measure [ 59 , 82 ], including lags for predictors or outcomes [ 28 , 30 , 58 , 63 , 65 , 79 ], using weights [ 24 , 67 ] or alternative data sources [ 37 , 69 ], or using non-imputed data [ 41 ].

As the methods and not the findings are the main focus of the current review, and because generic checklists cannot discern the underlying quality in this application field (see also below), we opted to pool all reported findings together, regardless of individual study characteristics or particular outcome(s) used, and speak generally of positive and negative effects on health. For this summary we have adopted the 0.05-significance level and only considered results from multivariate analyses. Strictly birth-related factors are omitted since these potentially only relate to the group of infant mortality indicators and not to any of the other general population health measures.

Starting with the determinants most often studied, higher GDP levels [ 21 , 26 , 27 , 29 , 30 , 32 , 43 , 48 , 52 , 58 , 60 , 66 , 67 , 73 , 79 , 81 , 82 ], higher health [ 21 , 37 , 47 , 49 , 52 , 58 , 59 , 68 , 72 , 82 ] and social [ 20 , 21 , 26 , 38 , 79 ] expenditures, higher education [ 26 , 39 , 52 , 62 , 72 , 73 ], lower unemployment [ 60 , 61 , 66 ], and lower income inequality [ 30 , 42 , 53 , 55 , 73 ] were found to be significantly associated with better population health on a number of occasions. In addition to that, there was also some evidence that democracy [ 36 ] and freedom [ 50 ], higher work compensation [ 43 , 79 ], distributional orientation [ 54 ], cigarette prices [ 63 ], gross national income [ 22 , 72 ], labor productivity [ 26 ], exchange rates [ 32 ], marginal tax rates [ 79 ], vaccination rates [ 52 ], total fertility [ 59 , 66 ], fruit and vegetable [ 68 ], fat [ 52 ] and sugar consumption [ 52 ], as well as bigger depth of credit information [ 22 ] and percentage of civilian labor force [ 79 ], longer work leaves [ 41 , 58 ], more physicians [ 37 , 52 , 72 ], nurses [ 72 ], and hospital beds [ 79 , 82 ], and also membership in associations, perceived corruption and societal trust [ 48 ] were beneficial to health. Higher nitrous oxide (NO) levels [ 52 ], longer average hospital stay [ 48 ], deprivation [ 51 ], dissatisfaction with healthcare and the social environment [ 56 ], corruption [ 40 , 50 ], smoking [ 19 , 26 , 52 , 68 ], alcohol consumption [ 26 , 52 , 68 ] and illegal drug use [ 68 ], poverty [ 64 ], higher percentage of industrial workers [ 26 ], Gross Fixed Capital creation [ 66 ] and older population [ 38 , 66 , 79 ], gender inequality [ 22 ], and fertility [ 26 , 66 ] were detrimental.

It is important to point out that the above-mentioned effects could not be considered stable either across or within studies. Very often, statistical significance of a given covariate fluctuated between the different model specifications tried out within the same study [ 20 , 49 , 59 , 66 , 68 , 69 , 73 , 80 , 82 ], testifying to the importance of control variables and multivariate research (i.e., analyzing multiple independent variables simultaneously) in general. Furthermore, conflicting results were observed even with regards to the “core” determinants given special attention, so to speak, throughout this text. Thus, some studies reported negative effects of health expenditure [ 32 , 82 ], social expenditure [ 58 ], GDP [ 49 , 66 ], and education [ 82 ], and positive effects of income inequality [ 82 ] and unemployment [ 24 , 31 , 32 , 52 , 66 , 68 ]. Interestingly, one study [ 34 ] differentiated between temporary and long-term effects of GDP and unemployment, alluding to possibly much greater complexity of the association with health. It is also worth noting that some gender differences were found, with determinants being more influential for males than for females, or only having statistically significant effects for male health [ 19 , 21 , 28 , 34 , 36 , 37 , 39 , 64 , 65 , 69 ].

The purpose of this scoping review was to examine recent quantitative work on the topic of multi-country analyses of determinants of population health in high-income countries.

Measuring population health via relatively simple mortality-based indicators still seems to be the state of the art. What is more, these indicators are routinely considered one at a time, instead of, for example, employing existing statistical procedures to devise a more general, composite, index of population health, or using some of the established indices, such as disability-adjusted life expectancy (DALE) or quality-adjusted life expectancy (QALE). Although strong arguments for their wider use were already voiced decades ago [ 84 ], such summary measures surface only rarely in this research field.

On a related note, the greater data availability and accessibility that we enjoy today does not automatically equate to data quality. Nonetheless, this is routinely assumed in aggregate level studies. We almost never encountered a discussion on the topic. The non-mundane issue of data missingness, too, goes largely underappreciated. With all recent methodological advancements in this area [ 85 – 88 ], there is no excuse for ignorance; and still, too few of the reviewed studies tackled the matter in any adequate fashion.

Much optimism can be gained considering the abundance of different determinants that have attracted researchers’ attention in relation to population health. We took on a visual approach with regards to these determinants and presented a graph that links spatial distances between determinants with frequencies of being studies together. To facilitate interpretation, we grouped some variables, which resulted in some loss of finer detail. Nevertheless, the graph is helpful in exemplifying how many effects continue to be studied in a very limited context, if any. Since in reality no factor acts in isolation, this oversimplification practice threatens to render the whole exercise meaningless from the outset. The importance of multivariate analysis cannot be stressed enough. While there is no “best method” to be recommended and appropriate techniques vary according to the specifics of the research question and the characteristics of the data at hand [ 89 – 93 ], in the future, in addition to abandoning simplistic univariate approaches, we hope to see a shift from the currently dominating fixed effects to the more flexible random/mixed effects models [ 94 ], as well as wider application of more sophisticated methods, such as principle component regression, partial least squares, covariance structure models (e.g., structural equations), canonical correlations, time-series, and generalized estimating equations.

Finally, there are some limitations of the current scoping review. We searched the two main databases for published research in medical and non-medical sciences (PubMed and Web of Science) since 2013, thus potentially excluding publications and reports that are not indexed in these databases, as well as older indexed publications. These choices were guided by our interest in the most recent (i.e., the current state-of-the-art) and arguably the highest-quality research (i.e., peer-reviewed articles, primarily in indexed non-predatory journals). Furthermore, despite holding a critical stance with regards to some aspects of how determinants-of-health research is currently conducted, we opted out of formally assessing the quality of the individual studies included. The reason for that is two-fold. On the one hand, we are unaware of the existence of a formal and standard tool for quality assessment of ecological designs. And on the other, we consider trying to score the quality of these diverse studies (in terms of regional setting, specific topic, outcome indices, and methodology) undesirable and misleading, particularly since we would sometimes have been rating the quality of only a (small) part of the original studies—the part that was relevant to our review’s goal.

Our aim was to investigate the current state of research on the very broad and general topic of population health, specifically, the way it has been examined in a multi-country context. We learned that data treatment and analytical approach were, in the majority of these recent studies, ill-equipped or insufficiently transparent to provide clarity regarding the underlying mechanisms of population health in high-income countries. Whether due to methodological shortcomings or the inherent complexity of the topic, research so far fails to provide any definitive answers. It is our sincere belief that with the application of more advanced analytical techniques this continuous quest could come to fruition sooner.

Supporting information

S1 checklist. preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (prisma-scr) checklist..

https://doi.org/10.1371/journal.pone.0239031.s001

S1 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s002

S2 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s003

• View Article
• Google Scholar
• PubMed/NCBI
• 75. Dahlgren G, Whitehead M. Policies and Strategies to Promote Equity in Health. Stockholm, Sweden: Institute for Future Studies; 1991.
• 76. Brunner E, Marmot M. Social Organization, Stress, and Health. In: Marmot M, Wilkinson RG, editors. Social Determinants of Health. Oxford, England: Oxford University Press; 1999.
• 77. Najman JM. A General Model of the Social Origins of Health and Well-being. In: Eckersley R, Dixon J, Douglas B, editors. The Social Origins of Health and Well-being. Cambridge, England: Cambridge University Press; 2001.
• 85. Carpenter JR, Kenward MG. Multiple Imputation and its Application. New York: John Wiley & Sons; 2013.
• 86. Molenberghs G, Fitzmaurice G, Kenward MG, Verbeke G, Tsiatis AA. Handbook of Missing Data Methodology. Boca Raton: Chapman & Hall/CRC; 2014.
• 87. van Buuren S. Flexible Imputation of Missing Data. 2nd ed. Boca Raton: Chapman & Hall/CRC; 2018.
• 88. Enders CK. Applied Missing Data Analysis. New York: Guilford; 2010.
• 89. Shayle R. Searle GC, Charles E. McCulloch. Variance Components: John Wiley & Sons, Inc.; 1992.
• 90. Agresti A. Foundations of Linear and Generalized Linear Models. Hoboken, New Jersey: John Wiley & Sons Inc.; 2015.
• 91. Leyland A. H. (Editor) HGE. Multilevel Modelling of Health Statistics: John Wiley & Sons Inc; 2001.
• 92. Garrett Fitzmaurice MD, Geert Verbeke, Geert Molenberghs. Longitudinal Data Analysis. New York: Chapman and Hall/CRC; 2008.
• 93. Wolfgang Karl Härdle LS. Applied Multivariate Statistical Analysis. Berlin, Heidelberg: Springer; 2015.

Mental Health Counseling

• Find Articles
• Search Strategies
• Find Journals
• Web Resources
• Qualitative vs. Quantitative Research
• Evidence Based Practice Research
• Citation: APA

Mixed Methods Research

As its name suggests, mixed methods research involves using elements of both quantitative and qualitative research methods. Using mixed methods, a researcher can more fully explore a research question and provide greater insight. 

Need to find quantitative or qualitative research?

The CINAHL and PsycINFO databases both allow for the application of filters that will yield results that are either qualitative or quantitative in nature. 

For detailed information about how to do that in CINAHL  or PsycINFO, visit the Quantitative and Qualitative LibGuide found here.   

What is Qualitative Research?

Quantitative research gathers data that can be measured numerically and analyzed mathematically. Quantitative research attempts to answer research questions through the quantification of data. 

Indicators of quantitative research include:

contains statistical analysis 

large sample size 

objective - little room to argue with the numbers 

types of research: descriptive studies, exploratory studies, experimental studies, explanatory studies, predictive studies, clinical trials 

What is Quantitative Research?

Qualitative research is based upon data that is gathered by observation. Qualitative research articles will attempt to answer questions that cannot be measured by numbers but rather by perceived meaning. Qualitative research will likely include interviews, case studies, ethnography, or focus groups. 

Indicators of qualitative research include:

interviews or focus groups 

small sample size 

subjective - researchers are often interpreting meaning 

methods used: phenomenology, ethnography, grounded theory, historical method, case study 

• << Previous: Web Resources
• Next: Evidence Based Practice Research >>
• Last Updated: Mar 27, 2024 2:05 PM
• URL: https://libguides.hofstra.edu/MentalHealthCounseling

This site is compliant with the W3C-WAI Web Content Accessibility Guidelines HOFSTRA UNIVERSITY Hempstead, NY 11549-1000 (516) 463-6600 © 2000-2009 Hofstra University

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Topic collections
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 13, Issue 2
• Qualitative Research Methods in Mental Health
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Sarah Peters
• Correspondence to : Dr Sarah Peters, School of Psychological Sciences, The University of Manchester, Coupland Building 1, Oxford Road M13 9PL, UK; sarah.peters{at}manchester.ac.uk

As the evidence base for the study of mental health problems develops, there is a need for increasingly rigorous and systematic research methodologies. Complex questions require complex methodological approaches. Recognising this, the MRC guidelines for developing and testing complex interventions place qualitative methods as integral to each stage of intervention development and implementation. However, mental health research has lagged behind many other healthcare specialities in using qualitative methods within its evidence base. Rigour in qualitative research raises many similar issues to quantitative research and also some additional challenges. This article examines the role of qualitative methods within mental heath research, describes key methodological and analytical approaches and offers guidance on how to differentiate between poor and good quality qualitative research.

https://doi.org/10.1136/ebmh.13.2.35

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

The trajectory of qualitative methods in mental health research

Qualitative methodologies have a clear home within the study of mental health research. Early and, arguably, seminal work into the study of mental illnesses and their management was based on detailed observation, moving towards theory using inductive reasoning. Case studies have been long established in psychiatry to present detailed analysis of unusual cases or novel treatments. Participant observation was the principle method used in Goffman's seminal study of psychiatric patients in asylums that informed his ideas about the institutionalising and medicalising of mental illness by medical practice. 1 However, the 20th century saw the ‘behaviourist revolution’, a movement where quantification and experimentation dominated. Researchers sought to identify cause and effects, and reasoning became more deductive – seeking to use data to confirm theory. The study of health and illness was determined by contemporary thinking about disease, taking a biomedical stance. Psychologists and clinical health researchers exploited natural science methodologies, attempting to measure phenomenon in their smallest entities and do so as objectively as possible. This reductionist and positivist philosophy shaped advances in research methods and meant that qualitative exploration failed to develop as a credible scientific approach. Indeed, ‘objectivity’ and the ‘discovery of truth’ have become synonymous with ‘scientific enquiry’ and qualitative methods are easily dismissed as ‘anecdotal’. The underlying epistemology of this approach chimes well with medical practice for which training is predominately in laboratory and basic sciences (such as physics and chemistry) within which the discourse of natural laws dominate. To this end, research in psychiatry still remains overwhelmingly quantitative. 2

Underlying all research paradigms are assumptions. However, most traditional researchers remain unaware of these until they start to use alternative paradigms. Key assumptions of quantitative research are that facts exist that can be quantified and measured and that these should be examined, as far as possible, objectively, partialling out or controlling for the context within which they exist. There are research questions within mental health where this approach can hold: where phenomenon of interest can be reliably and meaningfully quantified and measured, it is feasible to use data to test predictions and examine change. However, for many questions these assumptions prove unsatisfying. It is often not possible or desirable to try and create laboratory conditions for the research; indeed it would be ecologically invalid to do so. For example, to understand the experience of an individual who has been newly diagnosed with schizophrenia, it is clearly important to consider the context within which they live, their family, social grouping and media messages they are exposed to. Table 1 depicts the key differences between the two methodological approaches and core underlying assumptions for each.

• View inline

Comparison of underlying assumptions of quantitative and qualitative research approaches

It should be cautioned that it is easy to fall into the trap of categorising studies as either quantitative or qualitative. The two traditions are often positioned within the literature as opposing and in conflict. This division is unhelpful and likely to impede methodological advancement. Though, undeniably, there are differences in the two approaches to research, there are also many exceptions that expose this dichotomy to be simplistic: some qualitative studies seek to test a priori hypotheses, and some quantitative studies are atheoretical and exploratory. 3 Hence it is more useful to consider research methodologies as lying along a spectrum and that researchers should be familiar with the full range of methodologies, so that a method is chosen according to the research question rather than the researcher's ability.

Rationale for qualitative methods in current mental health research

There are a number of scientific, practical and ethical reasons why mental health is an area that can particularly benefit from qualitative enquiry. Mental health research is complex. Health problems are multifactorial in their aetiology and the consequences they have on the individual, families and societies. Management can involve self-help, pharmacological, educative, social and psychotherapeutic approaches. Services involved are often multidisciplinary and require liaison between a number of individuals including professionals, service-users and relatives. Many problems are exacerbated by poor treatment compliance and lack of access to, or engagement with, appropriate services. 4

Engagement with mental health research can also be challenging. Topics may be highly sensitive or private. Individuals may have impaired capacity or be at high risk. During the research process there may be revelations of suicidal ideation or criminal activity. Hence mental health research can raise additional ethical issues. In other cases scepticism of services makes for reluctant research participants. However, if we accept the case that meaningful research can be based in subjective enquiry then qualitative methods provide a way of giving voice to participants. Qualitative methods offer an effective way of involving service-users in developing interventions for mental health problems 5 ensuring that the questions asked are meaningful to individuals. This may be particularly beneficial if participants are stakeholders, for example potential users of a new service.

Qualitative methods are valuable for individuals who have limited literacy skills who struggle with pencil and paper measures. For example qualitative research has proved fruitful in understanding children's concepts of mental illness and associated services. 6

How qualitative enquiry is used within mental health research

There are a range of types of research question where qualitative methods prove useful – from the development and testing of theory, to the piloting and establishing efficacy of treatment approaches, to understanding issues around translation and implementation into routine practice. Each is discussed in turn.

Development and testing of theory

Qualitative methods are important in exploratory work and in generating understanding of a phenomenon, stimulating new ideas or building new theory. For example, stigma is a concept that is recognised as a barrier to accessing services and also an added burden to mental health. A focus-group study sought to understand the meaning of stigma from the perspectives of individuals with schizophrenia, their relatives and health professionals. 7 From this they developed a four-dimensional theory which has subsequently informed interventions to reduce stigma and discrimination that target not only engagement with psychiatric services but also interactions with the public and work. 7

Development of tools and measures

Qualitative methods access personal accounts, capturing how individuals talk about a lived experience. This can be invaluable for designing new research tools. For example, Mavaddat and colleagues used focus groups with 56 patients with severe or common mental health problems to explore their experiences of primary care management. 8 Nine focus groups were conducted and analysis identified key themes. From these, items were generated to form a Patient Experience Questionnaire, of which the psychometric properties were subsequently examined quantitatively in a larger sample. Not only can dimensions be identified, the rich qualitative data provide terminology that is meaningful to service users that can then be incorporated into question items.

Development and testing of interventions

As we have seen, qualitative methods can inform the development of new interventions. The gold-standard methodology for investigating treatment effectiveness is the randomised controlled trial (RCT), with the principle output being an effect size or demonstration that the primary outcome was significantly improved for participants in the intervention arm compared with those in the control/comparison arm. Nevertheless, what will be familiar for researchers and clinicians involved in trials is that immense research and clinical learning arises from these substantial, often lengthy and expensive research endeavours. Qualitative methods provide a means to empirically capture these lessons, whether they are about recruitment, therapy training/supervision, treatment delivery or content. These data are essential to improve the feasibility and acceptability of further trials and developing the intervention. Conducting qualitative work prior to embarking on an RCT can inform the design, delivery and recruitment, as well as engage relevant stakeholders early in the process; all of these can prevent costly errors. Qualitative research can also be used during a trial to identify reasons for poor recruitment: in one RCT, implementing findings from this type of investigation led to an increased randomisation rate from 40% to 70%. 9

Nesting qualitative research within a trial can be viewed as taking out an insurance policy as data are generated which can later help explain negative or surprising findings. A recent trial of reattribution training for GPs to manage medically unexplained symptoms demonstrated substantial improvements in GP consultation behaviour. 10 However, effects on clinical outcomes were counterintuitive. A series of nested qualitative studies helped shed light as to why this was the case: patients' illness models were complex, and they resisted engaging with GPs (who they perceived as having more simplistic and dualistic understanding) because they were anxious it would lead to non-identification or misdiagnosis of any potential future disease 11 , an issue that can be addressed in future interventions. Even if the insights are unsurprising to those involved in the research, the data collected have been generated systematically and can be subjected to peer review and disseminated. For this reason, there is an increasing expectation from funding bodies that qualitative methodologies are integral to psychosocial intervention research.

Translation and implementation into clinical practice

Trials provide limited information about how treatments can be implemented into clinical practice or applied to another context. Psychological interventions are more effective when delivered within trial settings by experts involved in their development than when they are delivered within clinical settings. 12 Qualitative methods can help us understand how to implement research findings into routine practice. 13

Understanding what stakeholders value about a service and what barriers exist to its uptake is another evidence base to inform clinicians' practice. Relapse prevention is an effective psychoeducation approach that helps individuals with bipolar disorder extend time to relapse. Qualitative methodologies identified which aspects of the intervention service-users and care-coordinators value, and hence, are likely to utilise in routine care. 14 The intervention facilitated better understanding of bipolar disorder (by both parties), demonstrating, in turn, a rationale for medication. Patients discovered new, empowering and less socially isolated ways of managing their symptoms, which had important impacts on interactions with healthcare staff and family members. Furthermore, care-coordinators' reported how they used elements of the intervention when working with clients with other diagnoses. The research also provided insights as to where difficulties may occur when implementing a particular intervention into routine care. For example, for care-coordinators this proved a novel way of working with clients that was more emotionally demanding, thus highlighting the need for supervision and managerial support. 14

Beginners guide to qualitative approaches: one size doesn't fit all

Just as there is a range of quantitative research designs and statistical analyses to choose from, so there are many types of qualitative methods. Choosing a method can be daunting to an inexperienced or beginner-level qualitative researcher, for it requires engaging with new terms and ways of thinking about knowledge. The following summary sets out analytic and data-generation approaches that are used commonly in mental health research. It is not intended to be comprehensive and is provided only as a point of access/familiarisation to researchers less familiar with the literature.

Data generation

Qualitative data are generated in several ways. Most commonly, researchers seek a sample and conduct a series of individual in-depth interviews, seeking participants' views on topics of interest. Typically these last upwards of 45 min and are organised on the basis of a schedule of topics identified from the literature or pilot work. This does not act as a questionnaire, however; rather, it acts as a flexible framework for exploring areas of interest. The researcher combines open questions to elicit free responses, with focused questions for probing and prompting participants to provide effective responses. Usually interviews are audio-recorded and transcribed verbatim for subsequent analysis.

As interviews are held in privately, and on one-to-one basis, they provide scope to develop a trusting relationship so that participants are comfortable disclosing socially undesirable views. For example, in a study of practice nurses views of chronic fatigue syndrome, some nurses described patients as lazy or illegitimate – a view that challenges the stereotype of a nursing professional as a sympathetic and caring person. 15 This gives important information about the education and supervision required to enable or train general nurses to ensure that they are capable of delivering psychological interventions for these types of problems.

Alternatively, groups of participants are brought together for a focus group, which usually lasts for 2 hours. Although it is tempting to consider focus groups as an efficient way of acquiring data from several participants simultaneously, there are disadvantages. They are difficult to organise for geographically dispersed or busy participants, and there are compromises to confidentiality, particularly within ‘captive’ populations (eg, within an organisation individuals may be unwilling to criticise). Group dynamics must be considered; the presence of a dominant or self-professed expert can inhibit the group and, therefore, prevent useful data generation. When the subject mater is sensitive, individuals may be unwilling to discuss experiences in a group, although it often promotes a shared experience that can be empowering. Most of these problems are avoided by careful planning of the group composition and ensuring the group is conducted by a highly skilled facilitator. Lester and colleagues 16 used focus-group sessions with patients and health professionals to understand the experience of dealing with serious mental illness. Though initially participants were observed via focus-group sessions that used patient-only and health professional only groups, subsequently on combined focus groups were used that contained both patients and health professionals. 16 The primary advantage of focus groups is that they enable generation of data about how individuals discuss and interact about a phenomenon; thus, a well-conducted focus group can be an extremely rich source of data.

A different type of data are naturally occurring dialogue and behaviours. These may be recorded through observation and detailed field notes (see ethnography in Table 2 ) or analysed from audio/ video-recordings. Other data sources include texts, for example, diaries, clinical notes, Internet blogs and so on. Qualitative data can even be generated through postal surveys. We thematically analysed responses to an open-ended question set within a survey about medical educators' views of behavioural and social sciences (BSS). 17 From this, key barriers to integrating BSS within medical training were identified, which included an entrenched biomedical mindset. The themes were analysed in relation to existing literature and revealed that despite radical changes in medical training, the power of the hidden curriculum persists. 17

Key features of a range of analytical approaches used within mental health research

Analysing qualitative data

Researchers bring a wide range of analytical approaches to the data. A comprehensive and detailed discussion of the philosophy underlying different methods is beyond the scope of this paper; however, a summary of the key analytical approaches used in mental health research are provided in Table 2 . An illustrative example is provided for each approach to offer some insight into the commonalities and differences between methodologies. The procedure for analysis for all methods involves successive stages of data familiarisation/immersion, followed by seeking and reviewing patterns within the data, which may then be defined and categorized as specific themes. Researchers move back and forth between data generation and analysis, confirming or disconfirming emerging ideas. The relationship of the analysis to theory-testing or theory-building depends on the methodology used.

Some approaches are more common in healthcare than others. Interpretative phenomenological (lPA) analysis and thematic analysis have proved particularly popular. In contrast, ethnographic research requires a high level of researcher investment and reflexivity and can prove challenging for NHS ethic committees. Consequently, it remains under used in healthcare research.

Recruitment and sampling

Quantitative research is interested in identifying the typical, or average. By contrast, qualitative research aims to discover and examine the breadth of views held within a community. This includes extreme or deviant views and views that are absent. Consequently, qualitative researchers do not necessarily (though in some circumstances they may) seek to identify a representative sample. Instead, the aim may be to sample across the range of views. Hence, qualitative research can comment on what views exist and what this means, but it is not possible to infer the proportions of people from the wider population that hold a particular view.

However, sampling for a qualitative study is not any less systematic or considered. In a quantitative study one would take a statistical approach to sampling, for example, selecting a random sample or recruiting consecutive referrals, or every 10th out-patient attendee. Qualitative studies, instead, often elect to use theoretical means to identify a sample. This is often purposive; that is, the researcher uses theoretical principles to choose the attributes of included participants. Healey and colleagues conducted a study to understand the reasons for individuals with bipolar disorder misusing substances. 18 They sought to include participants who were current users of each substance group, and the recruitment strategy evolved to actively target specific cases.

Qualitative studies typically use far smaller samples than quantitative studies. The number varies depending on the richness of the data yielded and the type of analytic approach that can range from a single case to more than 100 participants. As with all research, it is unethical to recruit more participants than needed to address the question at hand; a qualitative sample should be sufficient for thematic saturation to be achieved from the data.

Ensuring that findings are valid and generalisable

A common question from individuals new to qualitative research is how can findings from a study of few participants be generalised to the wider population? In some circumstances, findings from an individual study (quantitative or qualitative) may have limited generalisability; therefore, more studies may need to be conducted, in order to build local knowledge that can then be tested or explored across similar groups. 4 However, all qualitative studies should create new insights that have theoretical or clinical relevance which enables the study to extend understanding beyond the individual participants and to the wider population. In some cases, this can lead to generation of new theory (see grounded theory in Table 2 ).

Reliability and validity are two important ways of ascertaining rigor in quantitative research. Qualitative research seeks to understand individual construction and, by definition, is subjective. It is unlikely, therefore, that a study could ever be repeated with exactly the same circumstances. Instead, qualitative research is concerned with the question of whether the findings are trustworthy; that is, if the same circumstances were to prevail, would the same conclusions would be drawn?

There are a number of ways to maximise trustworthiness. One is triangulation, of which there are three subtypes. Data triangulation involves using data from several sources (eg, interviews, documentation, observation). A research team may include members from different backgrounds (eg, psychology, psychiatry, sociology), enabling a range of perspectives to be used within the discussion and interpretation of the data. This is termed researcher triangulation . The final subtype, theoretical triangulation, requires using more than one theory to examine the research question. Another technique to establish the trustworthiness of the findings is to use respondent validation. Here, the final or interim analysis is presented to members of the population of interest to ascertain whether interpretations made are valid.

An important aspect of all qualitative studies is researcher reflexivity. Here researchers consider their role and how their experience and knowledge might influence the generation, analysis and interpretation of the data. As with all well-conducted research, a clear record of progress should be kept – to enable scrutiny of recruitment, data generation and development of analysis. However, transparency is particularly important in qualitative research as the concepts and views evolve and are refined during the process.

Judging quality in qualitative research

Within all fields of research there are better and worse ways of conducting a study, and range of quality in mental health qualitative research is variable. Many of the principles for judging quality in qualitative research are the same for judging quality in any other type of research. However, several guidelines have been developed to help readers, reviewers and editors who lack methodological expertise to feel more confident in appraising qualitative studies. Guidelines are a prerequisite for the relatively recent advance of methodologies for systematic reviewing of qualitative literature (see meta-synthesis in Table 2 ). Box 1 provides some key questions that should be considered while studying a qualitative report.

Box 1 Guidelines for authors and reviewers of qualitative research (adapted from Malterud 35 )

▶ Is the research question relevant and clearly stated?

Reflexivity

▶ Are the researcher's motives and background presented?

Method, sampling and data collection

▶ Is a qualitative method appropriate and justified?

▶ Is the sampling strategy clearly described and justified?

▶ Is the method for data generation fully described

▶ Are the characteristics of the sample sufficiently described?

Theoretical framework

▶ Was a theoretical framework used and stated?

▶ Are the principles and procedures for data organisation and analysis described and justified?

▶ Are strategies used to test the trustworthiness of the findings?

▶ Are the findings relevant to the aim of the study?

▶ Are data (e.g. quotes) used to support and enrich the findings?

▶ Are the conclusions directly linked to the study? Are you convinced?

▶ Do the findings have clinical or theoretical value?

▶ Are findings compared to appropriate theoretical and empirical literature?

▶ Are questions about the internal and external validity and reflexivity discussed?

▶ Are shortcomings of the design, and the implications these have on findings, examined?

▶ Are clinical/theoretical implications of the findings made?

Presentation

▶ Is the report understandable and clearly contextualised?

▶ Is it possible to distinguish between the voices of informants and researchers?

▶ Are sources from the field used and appropriately referenced?

Conclusions and future directions

Qualitative research has enormous potential within the field of mental health research, yet researchers are only beginning to exploit the range of methods they use at each stage of enquiry. Strengths of qualitative research primarily lie in developing theory and increasing understanding about effective implementation of treatments and how best to support clinicians and service users in managing mental health problems. An important development in the field is how to integrate methodological approaches to address questions. This raises a number of challenges, such as how to integrate textual and numerical data and how to reconcile different epistemologies. A distinction can be made between mixed- method design (eg, quantitative and qualitative data are gathered and findings combined within a single or series of studies) and mixed- model study, a pragmatist approach, whereby aspects of qualitative and quantitative research are combined at different stages during a research process. 19 Qualitative research is still often viewed as only a support function or as secondary to quantitative research; however, this situation is likely to evolve as more researchers gain a broader skill set.

Though it is undeniable that there has been a marked increase in the volume and quality of qualitative research published within the past two decades, mental health research has been surprisingly slow to develop, compared to other disciplines e.g. general practice and nursing, with relatively fewer qualitative research findings reaching mainstream psychiatric journals. 2 This does not appear to reflect overall editorial policy; however, it may be partly due to the lack of confidence on the part of editors and reviewers while identifying rigorous qualitative research data for further publication. 20 However, the skilled researcher should no longer find him or herself forced into a position of defending a single-methodology camp (quantitative vs qualitative), but should be equipped with the necessary methodological and analytical skills to study and interpret data and to appraise and interpret others' findings from a full range of methodological techniques.

• Crawford MJ ,
• Cordingley L
• Dowrick C ,
• Edwards S ,
• ↵ MRC Developing and Evaluating Complex Interventions 2008
• Nelson ML ,
• Quintana SM
• Schulze B ,
• Angermeyer MC
• Mavaddat N ,
• Lester HE ,
• Donovan J ,
• Morriss R ,
• Barkham M ,
• Stiles WB ,
• Connell J ,
• Chew-Graham C ,
• England E ,
• Kinderman P ,
• Tashakkoria A ,
• Ritchie J ,
• Ssebunnya J ,
• Chilvers R ,
• Glasman D ,
• Finlay WM ,
• Strauss A ,
• Hodges BD ,
• Dobransky K
• Dixon-Woods M ,
• Fitzpatrick R ,
• Espíndola CR ,

Read the full text or download the PDF:

Transforming the understanding and treatment of mental illnesses.

Información en español

Celebrating 75 Years! Learn More >>

• Science News
• Meetings and Events
• Social Media
• Press Resources
• Email Updates
• Innovation Speaker Series

Revolutionizing the Study of Mental Disorders

March 27, 2024 • Feature Story • 75th Anniversary

At a Glance:

• The Research Domain Criteria framework (RDoC) was created in 2010 by the National Institute of Mental Health.
• The framework encourages researchers to examine functional processes that are implemented by the brain on a continuum from normal to abnormal.
• This way of researching mental disorders can help overcome inherent limitations in using all-or-nothing diagnostic systems for research.
• Researchers worldwide have taken up the principles of RDoC.
• The framework continues to evolve and update as new information becomes available.

President George H. W. Bush proclaimed  the 1990s “ The Decade of the Brain  ,” urging the National Institutes of Health, the National Institute of Mental Health (NIMH), and others to raise awareness about the benefits of brain research.

“Over the years, our understanding of the brain—how it works, what goes wrong when it is injured or diseased—has increased dramatically. However, we still have much more to learn,” read the president’s proclamation. “The need for continued study of the brain is compelling: millions of Americans are affected each year by disorders of the brain…Today, these individuals and their families are justifiably hopeful, for a new era of discovery is dawning in brain research.”

Still, despite the explosion of new techniques and tools for studying the brain, such as functional magnetic resonance imaging (fMRI), many mental health researchers were growing frustrated that their field was not progressing as quickly as they had hoped.

For decades, researchers have studied mental disorders using diagnoses based on the Diagnostic and Statistical Manual of Mental Disorders (DSM)—a handbook that lists the symptoms of mental disorders and the criteria for diagnosing a person with a disorder. But, among many researchers, suspicion was growing that the system used to diagnose mental disorders may not be the best way to study them.

“There are many benefits to using the DSM in medical settings—it provides reliability and ease of diagnosis. It also provides a clear-cut diagnosis for patients, which can be necessary to request insurance-based coverage of healthcare or job- or school-based accommodations,” said Bruce Cuthbert, Ph.D., who headed the workgroup that developed NIMH’s Research Domain Criteria Initiative. “However, when used in research, this approach is not always ideal.”

Researchers would often test people with a specific diagnosed DSM disorder against those with a different disorder or with no disorder and see how the groups differed. However, different mental disorders can have similar symptoms, and people can be diagnosed with several different disorders simultaneously. In addition, a diagnosis using the DSM is all or none—patients either qualify for the disorder based on their number of symptoms, or they don’t. This black-and-white approach means there may be people who experience symptoms of a mental disorder but just miss the cutoff for diagnosis.

Dr. Cuthbert, who is now the senior member of the RDoC Unit which orchestrates RDoC work, stated that “Diagnostic systems are based on clinical signs and symptoms, but signs and symptoms can’t really tell us much about what is going on in the brain or the underlying causes of a disorder. With modern neuroscience, we were seeing that information on genetic, pathophysiological, and psychological causes of mental disorders did not line up well with the current diagnostic disorder categories, suggesting that there were central processes that relate to mental disorders that were not being reflected in DMS-based research.”

Road to evolution

Concerned about the limits of using the DSM for research, Dr. Cuthbert, a professor of clinical psychology at the University of Minnesota at the time, approached Dr. Thomas Insel (then NIMH director) during a conference in the autumn of 2008. Dr. Cuthbert recalled saying, “I think it’s really important that we start looking at dimensions of functions related to mental disorders such as fear, working memory, and reward systems because we know that these dimensions cut across various disorders. I think NIMH really needs to think about mental disorders in this new way.”

Dr. Cuthbert didn’t know it then, but he was suggesting something similar to ideas that NIMH was considering. Just months earlier, Dr. Insel had spearheaded the inclusion of a goal in NIMH’s 2008 Strategic Plan for Research to “develop, for research purposes, new ways of classifying mental disorders based on dimensions of observable behavior and neurobiological measures.”

Unaware of the new strategic goal, Dr. Cuthbert was surprised when Dr. Insel's senior advisor, Marlene Guzman, called a few weeks later to ask if he’d be interested in taking a sabbatical to help lead this new effort. Dr. Cuthbert soon transitioned into a full-time NIMH employee, joining the Institute at an exciting time to lead the development of what became known as the Research Domain Criteria (RDoC) Framework. The effort began in 2009 with the creation of an internal working group of interdisciplinary NIMH staff who identified core functional areas that could be used as examples of what research using this new conceptual framework looked like.

The workgroup members conceived a bold change in how investigators studied mental disorders.

“We wanted researchers to transition from looking at mental disorders as all or none diagnoses based on groups of symptoms. Instead, we wanted to encourage researchers to understand how basic core functions of the brain—like fear processing and reward processing—work at a biological and behavioral level and how these core functions contribute to mental disorders,” said Dr. Cuthbert.

This approach would incorporate biological and behavioral measures of mental disorders and examine processes that cut across and apply to all mental disorders. From Dr. Cuthbert’s standpoint, this could help remedy some of the frustrations mental health researchers were experiencing.

Around the same time the workgroup was sharing its plans and organizing the first steps, Sarah Morris, Ph.D., was a researcher focusing on schizophrenia at the University of Maryland School of Medicine in Baltimore. When she first read these papers, she wondered what this new approach would mean for her research, her grants, and her lab.

She also remembered feeling that this new approach reflected what she was seeing in her data.

“When I grouped my participants by those with and without schizophrenia, there was a lot of overlap, and there was a lot of variability across the board, and so it felt like RDoC provided the pathway forward to dissect that and sort it out,” said Dr. Morris.

Later that year, Dr. Morris joined NIMH and the RDoC workgroup, saying, “I was bumping up against a wall every day in my own work and in the data in front of me. And the idea that someone would give the field permission to try something new—that was super exciting.”

The five original RDoC domains of functioning were introduced to the broader scientific community in a series of articles published in 2010  .

To establish the new framework, the RDoC workgroup (including Drs. Cuthbert and Morris) began a series of workshops in 2011 to collect feedback from experts in various areas from the larger scientific community. Five workshops were held over the next two years, each with a different broad domain of functioning based upon prior basic behavioral neuroscience. The five domains were called:

• Negative valence (which included processes related to things like fear, threat, and loss)
• Positive valence (which included processes related to working for rewards and appreciating rewards)
• Cognitive processes
• Social processes
• Arousal and regulation processes (including arousal systems for the body and sleep).

At each workshop, experts defined several specific functions, termed constructs, that fell within the domain of interest. For instance, constructs in the cognitive processes domain included attention, memory, cognitive control, and others.

The result of these feedback sessions was a framework that described mental disorders as the interaction between different functional processes—processes that could occur on a continuum from normal to abnormal. Researchers could measure these functional processes in a variety of complementary ways—for example, by looking at genes associated with these processes, the brain circuits that implement these processes, tests or observations of behaviors that represent these functional processes, and what patients report about their concerns. Also included in the framework was an understanding that functional processes associated with mental disorders are impacted and altered by the environment and a person’s developmental stage.

Preserving momentum

Over time, the Framework continued evolving and adapting to the changing science. In 2018, a sixth functional area called sensorimotor processes was added to the Framework, and in 2019, a workshop was held to better incorporate developmental and environmental processes into the framework.;

Since its creation, the use of RDoC principles in mental health research has spread across the U.S. and the rest of the world. For example, the Psychiatric Ratings using Intermediate Stratified Markers project (PRISM)   , which receives funding from the European Union’s Innovative Medicines Initiative, is seeking to link biological markers of social withdrawal with clinical diagnoses using RDoC-style principles. Similarly, the Roadmap for Mental Health Research in Europe (ROAMER)   project by the European Commission sought to integrate mental health research across Europe using principles similar to those in the RDoC Framework.;

Dr. Morris, who has acceded to the Head of the RDoC Unit, commented: “The fact that investigators and science funders outside the United States are also pursuing similar approaches gives me confidence that we’ve been on the right pathway. I just think that this has got to be how nature works and that we are in better alignment with the basic fundamental processes that are of interest to understanding mental disorders.”

The RDoC framework will continue to adapt and change with emerging science to remain relevant as a resource for researchers now and in the future. For instance, NIMH continues to work toward the development and optimization of tools to assess RDoC constructs and supports data-driven efforts to measure function within and across domains.

“For the millions of people impacted by mental disorders, research means hope. The RDoC framework helps us study mental disorders in a different way and has already driven considerable change in the field over the past decade,” said Joshua A. Gordon, M.D., Ph.D., director of NIMH. “We hope this and other innovative approaches will continue to accelerate research progress, paving the way for prevention, recovery, and cure.”

Publications

Cuthbert, B. N., & Insel, T. R. (2013). Toward the future of psychiatric diagnosis: The seven pillars of RDoC. BMC Medicine , 11 , 126. https://doi.org/10.1186/1741-7015-11-126  

Cuthbert B. N. (2014). Translating intermediate phenotypes to psychopathology: The NIMH Research Domain Criteria. Psychophysiology , 51 (12), 1205–1206. https://doi.org/10.1111/psyp.12342  

Cuthbert, B., & Insel, T. (2010). The data of diagnosis: New approaches to psychiatric classification. Psychiatry , 73 (4), 311–314. https://doi.org/10.1521/psyc.2010.73.4.311  

Cuthbert, B. N., & Kozak, M. J. (2013). Constructing constructs for psychopathology: The NIMH research domain criteria. Journal of Abnormal Psychology , 122 (3), 928–937. https://doi.org/10.1037/a0034028  

Garvey, M. A., & Cuthbert, B. N. (2017). Developing a motor systems domain for the NIMH RDoC program.  Schizophrenia Bulletin , 43 (5), 935–936. https://doi.org/10.1093/schbul/sbx095  

Insel, T. (2013). Transforming diagnosis . http://www.nimh.nih.gov/about/director/2013/transforming-diagnosis.shtml

Kozak, M. J., & Cuthbert, B. N. (2016). The NIMH Research Domain Criteria initiative: Background, issues, and pragmatics. Psychophysiology , 53 (3), 286–297. https://doi.org/10.1111/psyp.12518  

Morris, S. E., & Cuthbert, B. N. (2012). Research Domain Criteria: Cognitive systems, neural circuits, and dimensions of behavior. Dialogues in Clinical Neuroscience , 14 (1), 29–37. https://doi.org/10.31887/DCNS.2012.14.1/smorris  

Sanislow, C. A., Pine, D. S., Quinn, K. J., Kozak, M. J., Garvey, M. A., Heinssen, R. K., Wang, P. S., & Cuthbert, B. N. (2010). Developing constructs for psychopathology research: Research domain criteria. Journal of Abnormal Psychology , 119 (4), 631–639. https://doi.org/10.1037/a0020909  

• Presidential Proclamation 6158 (The Decade of the Brain) 
• Research Domain Criteria Initiative website
• Psychiatric Ratings using Intermediate Stratified Markers (PRISM)  
• Roadmap for Mental Health Research in Europe (ROAMER)  

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Qualitative Methods in Mental Health Services Research

Lawrence a. palinkas.

School of Social Work, University of Southern California

Qualitative and mixed methods play a prominent role in mental health services research. However, the standards for their use are not always evident, especially for those not trained in such methods. This paper reviews the rationale and common approaches to using qualitative and mixed methods in mental health services and implementation research based on a review of the papers included in this special series along with representative examples from the literature. Qualitative methods are used to provide a “thick description” or depth of understanding to complement breadth of understanding afforded by quantitative methods, elicit the perspective of those being studied, explore issues that have not been well studied, develop conceptual theories or test hypotheses, or evaluate the process of a phenomenon or intervention. Qualitative methods adhere to many of the same principles of scientific rigor as quantitative methods, but often differ with respect to study design, data collection and data analysis strategies. For instance, participants for qualitative studies are usually sampled purposefully rather than at random and the design usually reflects an iterative process alternating between data collection and analysis. The most common techniques for data collection are individual semi-structured interviews, focus groups, document reviews, and participant observation. Strategies for analysis are usually inductive, based on principles of grounded theory or phenomenology. Qualitative methods are also used in combination with quantitative methods in mixed method designs for convergence, complementarity, expansion, development, and sampling. Rigorously applied qualitative methods offer great potential in contributing to the scientific foundation of mental health services research.

There is a rich tradition of using qualitative methods in mental health services research, most notably represented in the ethnographies of populations with mental health problems (e.g., Estroff, 1981 ; Hopper, 1988), and the institutions that serve them (e.g., Caudill, 1958 ; Goffman, 1961 ; Rhodes, 1991). Nevertheless, as in other areas of scientific research ( Kuhn, 1970 ; Patton, 2002 ), qualitative methods in mental health services research have long been regarded as being “unscientific”, largely due to a lack of understanding of and experience with such methods ( Robins, Ware, dosReis, Willging, Chung, & Lewis-Fernández, 2008 ; Hopper, 2008 , Scarpinati Rosso & Bäärnhielm, 2012 ). This perspective began to change in the last two decades with calls for more of an interdisciplinary perspective and a recognition that qualitative methods could offer more in terms of an understanding of the need for and delivery of health services in general ( Berwick, 2008 ) and mental health services in particular ( Hohmann, 1999 ; Slade & Priebe, 2001 ) than was available from the use of quantitative methods alone. Since that time, qualitative methods have increasingly been used in mental health services research, both as the primary or exclusive method of data collection and analysis (e.g., Brunette et al., 2008 ; Proctor et al., 2008 ; Ware et al., 1999 ), and increasingly when combined with quantitative methods in mixed method designs ( Robins et al., 2008 ; Palinkas, Horwitz, Chamberlain, Hurlburt, & Landsverk, 2011 ). In both instances, there have been concerted efforts to demonstrate the rigor applied to the collection and analysis of qualitative data as well as the scientific basis for qualitative methods, characteristics that are also valued in the use of quantitative methods. In addition, the unique value of qualitative methods to scientific inquiry and understanding of mental health services has become more evident.

The aim of this paper is to provide an overview of the use of qualitative and mixed methods in mental health services and implementation research by drawing from the examples of their use embodied in the other papers in this special series, as well as from the larger mental health services literature, and to offer some guidelines on how such methods can and should be used to maximize their potential and insure rigor in their application to addressing important questions related to the need for and delivery of mental health services.

Rationale for using qualitative methods

Qualitative methods represent an approach to understanding that does not require, or does not lend itself to, enumeration ( Bernard, 2002 ). They can be viewed as both an art and a science. As in other fields of inquiry, they have often been used in mental health services research to provide a “thick description” (Geertz, 1970) of phenomena by providing a depth of understanding to complement the breadth of understanding afforded by quantitative methods, aiding in the interpretation of results obtained from quantitative methods, and contextualizing phenomena of interest. Examples of the use of qualitative methods in mental health services research for this purpose include Rhodes' (1991) ethnography of an emergency psychiatric unit; a descriptive account of the way in which clinicians reported making treatment decisions, their beliefs about how decisions should be made, and barriers to making treatment decisions ( Simmons, Hetrick & Jorm, 2013 ); use of qualitative data to contextualize the outcomes evaluation of a quality improvement approach for implementing evidence-based employment services in specialty mental health clinics (Hamilton et al., 2013); and an examination of the context and intervening mechanisms of an RCT evaluating an intervention for shared care in mental health ( Byng, Norman, Redfern & Jones, 2008 ). In the papers included in this special series, Rodriguez, Southam-Gerow and O'Connor (in press) use qualitative methods to “localize” evidence-based practices by providing the necessary insight into the local context in which practices that have been evaluated for their “global” generalizability must be applied.

Another major reason for the use of qualitative methods is that they are ideal for eliciting the perspectives of those being studied. Qualitative methods “allow people to speak in their own voice, rather than conforming to categories and terms imposed on them by others” ( Sofaer, 1999 , p. 1105). By eliciting participant perspectives, qualitative methods serve to enhance the validity of data being collected because it enables the investigator to compare their own perception of reality with the perception of those who are being studied. For instance, Lee, Munson, Ware, Ollie, Scott, and McMillen (2006) gave youths in foster care an opportunity to voice their experiences with mental health services and specific providers. Turner, Sharp, Folkes and Chew-Graham (2008) conducted in-depth interviews to explore women's views and experiences of antidepressants as a treatment for postnatal depression. In this special series, Rodriguez and colleagues (in press) used qualitative methods explicitly to better understand the perspectives of three groups of stakeholders on children's mental health services (parents, clinicians, and clinical directors) with the intention of seeing how these perspectives compared with one another. Similarly, Lyon, Ludwig, Romano, Koltracht, Vander Stoep, and McCauley (2013) used qualitative methods to elicit provider perspectives on the appropriateness of implementing an evidence-based, modular therapeutic approach within a school-based health clinic. Dorsey, Conover and Cox (in press) sampled foster parents to elicit their perspectives on engagement that could be incorporated into the adaptation of an existing engagement intervention. Murray and colleagues (2013) assessed counselors, children and caregivers perspectives on the use of Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) in Zambia.

Qualitative methods are often found to be especially useful during initial stages of research because they enable investigators to acquire some understanding of issues, to obtain “pilot data”, or when there is too little previous research or absence of theory to allow for identification of hypotheses to be tested. Examples of such use include an exploration of the experiences of providers who have encountered immigrant patients in their services on a daily basis and to reflect on areas where difficulties may arise and how these are managed in mental health services (Sandu et al, 2012) and an exploration of wraparound services for youth with serious emotional disturbance ( Mendenhall et al., 2013 ). Chew-Graham, Slade, Montana, Stewart, and Gask (2007) used qualitative methods to explore the function of community health teams in managing referral decisions at the primary-specialist interface from the perspectives of both referrers (primary care providers) and referred to (mental health specialists). In this special series, Lyon and colleagues (2013) employed qualitative methods “because the study of modular psychotherapy is in its infancy.” Dorsey and colleagues (in press) used qualitative methods as part of a pilot project effort to tailor an existing engagement intervention for use in implementing TF-CBT with a small group of foster parents to identify any population-specific areas of adaptation to the engagement intervention. Such exploration can be used to develop new theories or conceptual frameworks or to expand upon existing ones and to generate new hypotheses that may be tested using quantitative methods or to develop valid and reliable quantitative methods by identifying the content and form of questions to be asked and by identifying the target population through observation and interviews. For instance, Byng and colleagues (2008) used qualitative methods to generate provisional hypotheses, ready to be tested using the analytic induction process, that were derived both from themes emerging during initial coding of interview transcripts and by examining the matrix of obvious patterns. Beehler, Funderburk, Possemato and Vair (2013) used qualitative methods to develop a self-report measure of behavioral health provider adherence to co-located, collaborative care.

Finally, qualitative methods have been used in mental health services research for an evaluation of process. Such methods are frequently used in evaluation research to explain how a program or intervention operates. Harris and colleagues (2012) used qualitative methods to help understand why Early Intervention in Psychosis (EIP) services are effective by exploring the personal experiences of a small sample of services users. Byng et al. (2008) used qualitative methods to conduct a process evaluation of a complex intervention for shared care in mental health. In the papers in this series, Aarons and colleagues (in press) use qualitative methods to examine the process of scaling up of an EBP across an entire service system using an Interagency Collaborative Team (ICT) approach. Such methods can provide insight as to why program does not work as intended; it may also provide insight into unanticipated benefits or outcomes.

Characteristics of Qualitative Methods

Qualitative and quantitative methods are similar in that they both adhere to certain principles of scientific inquiry and rigor. The principles of validity, reliability, generalizability and objectivity that govern sound quantitative research have their counterparts in the principles of credibility, dependability, transferability, and reflexivity that govern sound qualitative research ( Patton, 2002 ; Bernard, 2002 ). One of the techniques used to insure validity in qualitative methods is the concept of “saturation, the point at which no additional data collection in needed, no new codes are developed, and themes and subthemes have been fully fleshed out” ( Padgett, 2008 , p. 171). Saturation refers to completeness or fullness necessary to insure that everything related to the phenomenon of inquiry that can be collected and analyzed has been done so within the limits of the forms of collection and analysis chosen. In a study of women's attitudes toward using antidepressants to treat postnatal depression, Turner and colleagues (2008) reached saturation of key themes after interviews with 27 women, while Chew-Graham et al. (2007) achieved saturation with 35 general practitioners. Another technique used to insure validity is the identification of deviant or nonconfirmatory cases, the exceptions to the rule. This technique was employed by Turner and colleagues (2008) in their study of women's views of antidepressants as a treatment for postnatal depression and by Chew-Graham and colleagues (2007) in their study of community mental health teams. Validity has also been supported by means of “member checking” whereby study participants or others who share similar characteristics review study findings to confirm and potential elaborate on them. In their study, Dorsey and colleagues (in press) presented the findings of their interviews with foster parents to two boards, one comprised of foster parents and one comprised of caseworkers. They further “triangulated” the data collected from both boards.

Reliability or dependability of qualitative data analysis is usually achieved by establishing a specified level of agreement in identification of topics or themes through both qualitative and quantitative means. The inductive approach typically includes a process of “coding by consensus”, which includes one or both of two activities, 1) regular meetings among coders to discuss procedures for assigning codes to segments of data and resolve differences in coding procedures, and 2) comparison of codes assigned on selected transcripts to calculate a percent agreement or kappa measure of inter-rater reliability. Most studies in the mental health services literature report the first method (e.g., Gilburt, Slade, Bird, Oduola & Craig, 2013 ; Mittal, Drummond, Blevins, Curran, Corrigan, & Sullivan, 2013 ; Turner et al., 2008 ), while others report measures of inter-rater reliability of coding of qualitative transcripts (e.g., Bradley et al., 2003 ; Lee et al., 2007; Palinkas et al., 2008 ). Rodriguez and colleagues (in press) appear to have used both methods to insure the reliability of coding assigned; however, they also acknowledge that they relied primarily on a priori codes rather than identify new or emergent codes. (in press) and colleagues, Lyon and colleagues (2013) and Dorsey and colleagues (in press) rely only on the first method.

Qualitative methods also acknowledge the importance of generalizability, often referred to in the literature as the transferability of findings from one context or population to another. For the most part, this acknowledgment is usually described as a limitation. For instance, Rodriguez and colleagues (in press) note that all the parents recruited in their study were women. Lyon and colleagues (2013) urge caution when generalizing findings to other models or settings because their study included a sample of clinicians working in one particular type of school-based service delivery system. Murray and colleagues (2013) note their study was conducted only with children and caregivers of children who completed treatment and thus did not include the perspectives of children who did not complete treatment. It was also limited to local lay workers in an urban setting. However, as noted earlier, as qualitative methods are designed for depth and not breadth of understanding, the generalizability of findings is of less importance at this stage of the process of scientific inquiry than the attainment of maximum insight from the data that are collected.

Another characteristic of some qualitative studies is the explicit reflexivity employed by investigators as a means of identifying and addressing potential biases in the collection and interpretation of data. Such bias may be associated with the investigator's preconceived beliefs, assumptions and theoretical orientations; demographic characteristics; experience with the methods used and familiarity with the phenomenon under investigation. Gianakis and Carey (2011) , for instance, make explicit their background and initial expectations in their study of adults who experience improvement in psychological functioning without the benefit of psychotherapy.

Other methods used in qualitative studies to enhance rigor of analysis include triangulation of viewpoints by purposefully interviewing people in various roles within an organization, peer debriefing and support meetings among research team members, and providing a detailed audit trail during analysis ( Harris, Collinson, & das Nair, 2012 ; Miles & Huberman, 1994 ; Mendenhall, Kapp, Rand, Robbins, & Stipp, 2013 ).

Design Strategies

While adhering to the same scientific standards, qualitative methods are nevertheless distinguished from quantitative methods by certain other characteristics, including design strategies, data collection and analysis strategies. One of the most obvious distinctions in the design of qualitative studies is the reliance upon generally smaller samples for investigation. Although there are qualitative mental health services studies involving large samples, they generally do not require them since their aim is to achieve a depth of understanding rather than a breadth (generalizability) of understanding. Consequently, the number of participants is often based on availability of participants and feasibility of data collection rather than quantitative power calculations. Nevertheless, there are certain conventions in identifying how many participants to include in a qualitative study, including precedent and saturation ( Guest, Bunce & Johnson, 2006 ; Miles & Huberman, 1994 ; Strauss & Corbin, 1998 ).

Another familiar feature of qualitative study designs is the use of purposeful sampling to identify and recruit study participants. Unlike quantitative studies that rely on random samples to minimize bias and confounding, qualitative studies rely principally on “purposeful sampling” designed to maximize the information gained from what is typically a much smaller group of participants than found in most quantitative studies ( Palinkas, Horwitz, Green, Wisdom, Duan, & Hoagwood, 2013 ). Among the most common forms of purposeful sampling are extreme or deviant case sampling, criterion sampling and maximum variation sampling. Sampling extreme or deviant cases is designed to reduce variation and highlight the most prominent features of a phenomenon under investigation. For instance, Sandu and colleagues (2012) sampled services providers in 16 European countries after first sampling mental health services in areas with high concentration of immigrants in consultation with a research center for that country and then asking for an interview with a practitioner with the most extensive experience of providing mental health care to immigrants in the service. Criterion sampling is also intended to reduce the range of variation and limit the possibility of collecting information not directly related to the phenomenon of interest. For instance, Byng et al (2008) limited their sample to participants of an RCT of a complex health services intervention for shared care for people with long-term mental illness. As its name suggests, maximum variation sampling is intended to expand the range of variation and thereby select participants who are representative of a larger population and can maximize the opportunity for a comprehensive view of the phenomenon of interest. In a study of implementation issues related to several evidence-based practices for adults with serious mental illness that were included in a national demonstration project, Isett and colleagues (2007) asked the mental health commissioner's office in each participating state to identify potential participants who were knowledgeable about evidence-based practice (itself a criterion) and came from various backgrounds to capture a broad range of perspectives. Chew-Graham et al (2007) sampled general practitioners to insure variation in gender, ethnicity, experience, and practice size. Other forms of purposeful sampling have been used in mental health services research, including random sampling ( Mendenhall et al., 2013 ) and convenience sampling ( Mittal et al., 2013 ) or both ( Stergiopoulos et al., 2012 ), but these two in particular are usually considered less likely to obtain information rich participants ( Patton, 2002 ).

In this special series, Rodriguez and colleagues (in press) wanted to obtain a sample that was representative of the clinics that were the focus of their investigation using different sampling methods. They did so by sampling the universe of directors and clinic-affiliated therapists who worked in the study clinics and a subset of parents through informational flyers and invitations from providers. The other studies also appear to have used criterion sampling to identify potential participants, including full-time mental health provider in their respective schools ( Lyon et al., 2013 ), foster parents based on referrals from caseworkers and youth exposure to at least one traumatic event and youth PTS symptoms ( Dorsey et al., in press ), all “outer” and “inner” context stakeholders involved in the implementation of an evidence-based practice in one county ( Aarons et al., in press ) and participants in a larger feasibility study of TF-CBT in Zambia ( Murray et al., 2013 ). While these strategies are commonly employed in mental health services researchers, investigators should be mindful of the fact that representativeness is only one criterion for purposeful sampling. Other important criteria include familiarity with the topic under investigation based on experience, a willingness to share that information, and an ability to share that information based on the informant's own insight and communication skills ( Bernard, 2002 ).

Another feature of qualitative research design observed in mental health services research is its emphasis on naturalistic inquiry, “a ‘discovery-oriented’ approach that minimizes investigator manipulation of the study setting and places no prior constraints on what the outcomes of the research will be ( Patton, 2002 , p. 39). Qualitative designs are, for the most part, observational in nature. Data are collected in situ, usually as events happen. These qualities are more often employed in long-duration ethnographic studies that make use of participant observation than in focus groups and semi-structured interviews to collect information guided by a priori conceptual frameworks ( Padgett, 2008 ). The reason for this focus is to avoid or eliminate any potential bias, either on the part of the observer or those being observed or interviewed ( Guba, 1978 ).

A final characteristic of qualitative studies of mental health services is the emergent and iterative nature of qualitative research. The emergent design is based on the principle that circumstances often dictate changes in focus or means of data collection and that the researcher should be prepared to accommodate to those changes rather than adhere to a plan to use potentially inappropriate or inadequate methods ( Padgett, 2008 ). Qualitative mental health services research are often iterative in nature in which there is a constant back and forth between data collection and analysis ( Bernard, 2002 ). In contrast, quantitative studies generally initiate the analysis phase only after data collection is complete or near completion. For instance, Isett and colleagues (2007) developed a protocol for conducting follow-up interviews based on three dominant themes identified in their analysis of the first set of interviews. Chew-Graham et al (2007) modified their interview schedule in light of emerging data.

Data Collection and Fieldwork Strategies

There exist several different forms of data collection strategies in qualitative studies of mental health services. The most common strategies are extended interviews and focus groups, followed by ethnographic fieldwork, document reviews and more structured approaches that involve both qualitative and quantitative methods. The extended interview is the most frequently used method for data collection in mental health services research and is intended to elicit information on the participant's experience, opinions, and perceptions of mental health services. This form of data collection can range from brief responses to open-ended questions on more structured interviews or surveys (e.g., Marcus, Westra, Angus, Kertes , 2011 ) to a series of extended interviews with “key informants,” individuals especially knowledgeable about the topic under examination (e.g., Sandu et al., 2012). In this special series, Aarons and colleagues (in press) conducted interviews with executive staff from a state child welfare system, CBOs providing home visitation services, and a local foundation, using an interview guide consisting of open-ended questions tailored to each stakeholder group to assess roles and responsibilities and perceptions of the implementation of the SafeCare intervention. Lyon et al. (2013) conducted one-hour semi-structured interviews with 17 school-based mental health providers. Dorsey and colleagues (in press) conducted interviews of shorter duration (13 to 27 minutes) with 7 foster parents to collect information on the initial telephone call to facilitate engagement and experience with the first TF-CBT treatment session. Rodriguez et al. (in press) interviewed three parents using a semi-structured guide to obtain perceptions of causes of anxiety, depression and conduct-related problems in children; ideal treatments for these problems; barriers to making these treatments available; and additional comments. Murray et al (2013) trained counselors to ask a series of six open-ended questions to children in Zambia undergoing treatment for trauma and their caregivers, although they acknowledge potential problems with counselors interviewing children and caregivers, leading to hesitation to report negative feedback. This was followed by a second interview for further clarification and probing on the participants' initial responses.

A particular form of extended interviewing is the structured narrative, in which the participant describes the experience of having an illness and seeking services for that illness. Scarpinatti Rosso and Bäärnhielm (2012) collected narratives from 23 newly referred immigrants seeking help at a psychiatric outpatient clinic in Stockholm, Sweden, using a Cultural Formulation (CF) interview protocol ( Bäärnhielm, Scarpinatti Rosso, & Pattyi, 2009 ). Marcus et al (2011) used narratives to understand client experiences of using motivational interviewing for treatment of generalized anxiety disorder. Rappaport, Jerzembek, Doel, Jones, Cella and Lloyd (2010) used narrative content analysis ( Coffey & Atkinson, 1996 ) to construct narratives of uncertainties about treatment of mental health from free text responses to a questionnaire.

Another form of data collection that has been used extensively in mental health services research is the focus group. Focus groups are interviews that are designed to use group interaction to generate data and insights less accessible in individual interviews ( Krueger, 1988 ; Morgan, 1988 ). Although this cannot always be achieved in service settings, the ideal composition of a focus group is between 6 and 10 “homogeneous strangers,” individuals who are similar by virtue of their experience with or familiarity with the topic but who otherwise are not closely linked to one another. In the articles in this special series, Aarons and colleagues (in press) conducted 9 focus groups with case manager supervisors, trainers, the seed team, and service provider team trained by the seed team. Rodriguez and colleagues (in press) conducted two focus groups with 11 providers using a guide similar in structure to the one used with individual interviews with parents; however, they acknowledge that while perhaps logistically feasible, the focus group format “created an uncomfortable environment for staff, making it difficult to disclose in the presence of fellow colleagues”.

Ethnographic fieldwork often consists of several different modes of data collections, but perhaps the most distinctive feature is the technique of participant observation. Participant observation consists of spending time and talking with people in their own settings ( Ware et al., 1999 ). Estroff conducted ethnographic fieldwork with a group of discharged mental hospital patients living in Madison, Wisconsin. Her participation included taking antipsychotic medication to better understand the challenges of living with their side effects. Ware et al (1999) engaged in participant observation at two public community mental health centers (CMHCs) and one emergency psychiatric evaluation unit in Boston to identify the interpersonal processes of giving and receiving day-to-day services through which individual providers create experiences of continuity for consumers. Palinkas and colleagues (2008) participated in training workshops in three different evidence-based treatments whose effectiveness in their standard use and in a modular fashion in a randomized controlled trial. They also conducted site visits of each of the clinics participating in the study.

Finally, some qualitative mental health services studies have relied upon more quasi-statistical techniques for data collection. These techniques often represent the iterative nature of qualitative methods in that the investigators alternate between qualitative data collection, transformation of qualitative data into quantitative data, and validation or elaboration using another round of qualitative data collection. An illustration of this process is the technique of concept mapping ( Trochim, 1989 ). Aarons, Wells, Zagursky, Fettes, and Palinkas (2009) , solicited information on factors likely to impact implementation of EBPs in public sector mental health settings from 31 services providers and consumers organized into 6 focus groups. Each participant then sorted a series of 105 statements into piles and rated each statement according to importance and changeability. Data were then entered in a software program that uses multidimensional scaling and hierarchical cluster analysis to generate a visual display of how statements clustered across all participants. Finally, 22 of the original 31 participants assigned meaning to and identified an appropriate name for each of the clusters identified ( Aarons et al, 2009 ). Another technique for data collection that relies on the iterative collection and analysis of data is the Delphi approach where opinions from content experts are collected and summarized with the primary goal of consensus building, thereby helping to insure content validity. Beehler et al. (2013) used this technique to develop a list of self-report measures of behavioral health provider adherence to co-located, collaborative care, beginning with the development of a 56-item measure of collaborative care, obtaining qualitative feedback from content experts while quantitatively rating each item's relevance for behavioral health provider practice through three rounds of emailed surveys. Items with consensus ratings of 80 percent or greater were included in the final adherence measure.

Data Analysis Strategies

For the most part, qualitative studies rely on a variety of methods for inductive analysis and creative synthesis. For instance, Byng and colleagues analyzed their data using Realistic Evaluation, “a framework for a context sensitive process evaluation accompanying an RCT, designed to unpack the complexity of the intervention by examining interactions between intervention components and context and then further refining its core functions” (2008, pp. 3-4). This modified form of analytic induction was used to examine the empirical data from case studies and iteratively build “middle range theories.” However, there are instances of qualitative mental health services research that have also employed deductive approaches. For instance, in a study of stigma associated with PTSD among treatment seeking combat veterans, Mittal and colleagues (2013) began with an inductive approach based on grounded theory methods ( Strauss and Corbin, 1998 ), followed by a deductive analysis with the use of an a priori model of the participants reaction to the stigmatizing labels they perceived. Hamilton and colleagues (2013) used a hybrid deductive/inductive thematic analysis approach in their study of implementation of employment services in specialty mental health.

Many of the coding strategies employed for analyzing qualitative data in mental health services research fall under the general rubric of “content” or “thematic” analysis. Such analysis often involves a rigorous process of reviewing transcripts and other documents line by line and assigning codes based on a priori and/or emergent topics or themes, and the construction of themes ( Miles & Huberman, 1994 ; Strauss & Corbin, 1998 ). Coding also occurs in stages in which initial preliminary codes are followed by secondary or focused coding (e.g., Green et al., 2008 ; Simmons et al., 2013 ), or in which open codes are followed by axial codes (e.g., Hamilton et al., 2013). The papers by Rodriguez and colleagues (in press) , Lyon and colleagues (2013) , and Dorsey and colleagues (in press) provide an illustration of the inductive process or content or thematic analysis. They refer to the process of “unitizing” (i.e., construction of units) the data by creation of codes based on an a priori classification system (the MHSE model), construction of a codebook with a list of these units, and then the identification of themes through the use of text analysis software such as NVivo ( Dorsey et al., in press ; Rodriguez et al., in press ) or Atlas.ti ( Lyon et al., 2013 ). The papers by Murray et al (2013) and Aarons et al. (in press) employ another commonly used analytic process found in MHSR qualitative studies. Similar in many ways to the content analysis based on a priori topics described above, this process adheres more to a grounded theory ( Glaser & Strauss, 1967 ; Strauss & Corbin, 1998 ) in which both a priori and emergent topics are coded to construct a conceptual framework or theory. Stergiopoulos and colleagues (2012) analyzed interview and focus group transcripts of a Housing First model for homeless individuals with mental illness using a grounded theory methodology. Isett et al (2007) utilized grounded theory case study methods developed by Yin (2003) . In addition to systematizing the process of coding the data, qualitative analysts using this approach engage in an iterative process of “constant comparison” (e.g., Chew-Graham et al., 2007 ; Turner et al., 2008 ).

Although not as common as the grounded theory approach, another approach to qualitative data analysis used in mental health services is based in a phenomenological tradition. Drawing from the work of Husserl (1962) , Schutz (1970) , and others, phenomenology aims at gaining a deeper understanding of the nature or meaning of our everyday experiences. As equally concerned with rigor as is the grounded theory approach described above, phenomenology gives more attention to understanding the lived experience of individuals using or in need of mental health services while controlling for preconceptions and potential biases on the investigator. Gianakis and Carey (2011) utilized Interpretative Phenomenological Analysis (IPA: Smith, 1996 ), a cyclical iterative process with a constant revisiting of transcripts to insure the superordinate themes generated directly relate to the shared experience of the participants, in their investigation of the phenomenological experience of psychological change following distress from a range of problems in individuals who have not used psychotherapy to resolve those problems. Harris and colleagues (2012) used IPA to explore the experiences of being in contact with Early Intervention in Psychosis (EIP) services in a small sample of service users.

While the detail underlying the coding of data and generation of themes is critical to demonstrating the rigor applied to qualitative analysis, a holistic perspective is equally important. This perspective requires that the investigator goes beyond the enumeration of themes to provide “the big picture”, for instance, by explaining how the themes are linked together to provide a more comprehensive understanding of their meaning, operation, and relationships, and by paying particular attention to context. For instance, in the study by Lyon and colleagues (2013) , key elements of the fit between the modular approach and the school context at the client and clinician levels are summarized, elaborated and integrated using the identified themes. Aarons and colleagues (in press) relied on the principle of constant comparison to condense coding categories into broader themes using a format that placed the phenomenon under investigation within a broader framework of understanding collaborations, negotiations and resolutions while considering inner and outer contextual characteristics.

One analytical strategy used in mental health services research to provide such a holistic perspective is the case study approach. Often relying on multiple forms of qualitative data (interviews, focus groups, participant observation) rather than a single form, case studies are less concerned with representativeness or generalizability and more concerned with richness in detail of individuals, groups, organizations, systems or experiences and their context ( Yin, 2003 ). Examples include a multiple case study of implementation as usual in children's social service organizations ( Powell et al., 2013 ), a phenomenological case study of communication between clinicians about attention-deficit/hyperactivity disorder assessment ( Lynch, Cho, Ogle, Sellman & dosReis, 2014 ), and an evaluation of the state policy context of implementation of several evidence-based practices for adults with serious mental illness ( Isett et al., 2007 ).

Mixed Methods

Qualitative methods are increasingly represented in mental health services research in the form of mixed method designs that focus on collecting, analyzing and merging both quantitative and qualitative data into one or more studies. The central premise of these designs is that the use of quantitative and qualitative approaches in combination provides a better understanding of research issues than either approach alone (Cresswell & Plano Clark, 2011). In such designs, qualitative methods are used to explore and obtain depth of understanding while quantitative methods are used to test and confirm hypotheses and obtain breadth of understanding of the phenomenon of interest ( Teddlie & Tashakkori, 2003 ).

Mixed method designs in mental health services research can be categorized in terms of their structure, function, and operation ( Palinkas et al., 2011 ). Quantitative and qualitative methods may be used simultaneously or sequentially, with one method viewed as dominant or primary and the other as secondary, although equal weight can be given to both methods. The function of mixed method designs are usually based on whether the two methods are used to answer the same question or to answer related questions and whether they were used to achieve convergence (using both types of methods to answer the same question, either through comparison of results to see if they reach the same conclusion or by converting a data set from one type into another, e.g., quantifying qualitative data or qualifying quantitative data); complementarity (using each set of methods to answer a related question or series of questions for purposes of evaluation or elaboration, e.g., using qualitative data to examine treatment process and quantitative methods to examine treatment outcome); expansion (using one type of method to answer questions raised by the other type of method, e.g., using qualitative methods to explain findings from an analysis of quantitative data); development (using one type of method to answer questions that will enable use of the other method to answer other questions, e.g., using qualitative methods to construct a questionnaire or a theoretical model that can be tested using qualitative methods); or sampling (using one type of method to define or identify the participant sample for collection and analysis of data representing the other type of method, e.g., purposefully selecting participants for individual interviews based on their responses to a survey). Finally, the use of mixed methods in mental health services research involves three distinct processes or strategies for combining qualitative and quantitative data: merging or converging the two datasets by actually bringing them together, connecting the two datasets by having one build upon the other, or embedding one dataset within the other, so that one type of data provides a supportive role for the other dataset.

Some of the articles in this special series offer illustrations of the combining of qualitative data. For instance, Rodriquez and colleagues enumerated the number of units within each of the identified themes and compared these units using nonparametric statistics. Lyon et al. (2013) identified the percentage of participants who mentioned a particular topic or theme during their interview. The salience of the topics was indicated by the percentage of clinicians who discussed them during the interviews. Murray and colleagues (2013) also use frequency counts to indicate the salience or importance of identified themes. The study by Rodriguez et al (in press) was an early phase of a mixed-method university community partnership endeavor designed to adapt and test evidence-based practices for anxiety and depression. The study by Lyon et al (2013) occurred subsequent to an intervention study, while the study by Dorsey et al. (in press) occurred in the first phase of a two-phase feasibility trial of TF-CBT with youth in foster care.

The technique of concept mapping used by Aarons et al. (2009) , where qualitative data elicited from focus groups are “quantitized” using multidimensional scaling and hierarchical cluster analysis, is an example of convergence. In a study of the implementation of evidence-based psychotherapies for PTSD in VA specialty clinics, Watts and colleagues (2014) conducted semi-structured interviews with staff at participating clinics using the PARiHS framework to develop overarching questions. Transcripts of these interviews were then coded by domain and element of the PARiHS framework and then a scoring rubric was used to transform each element of the framework into a numeric value. They then conducted a Poisson linear regression that used element scores for each facility as independent variables and percentage of patients at each sites receiving any evidence-based therapy as the dependent variable. Gilburt et al. (2013) used mixed methods to achieve complementarity in their evaluation of implementation of a recovery-oriented practice through training across a system of mental health services, using a quantitative audit of care plans in a random sample of 700 patients to assess change in core plan topics and in responsibility of action and semi-structured interviews with team leaders to explore understanding of recovery, implementation within the service and the wider system, and perceived impact of the training on individual practice and that of the team. The Delphi approach used by Beehler and colleagues (2013) is an example of mixed methods to achieve development and elaboration. Woltmann et al (2008) used qualitative data obtained through interviews with staff, clinic directors and consultant trainers to create categories of staff turnover and designations of positive, negative and mixed influence of turnover on implementation outcomes. These categories were then quantitatively compared with implementation outcomes via simple tabulations of fidelity and penetration means for each category.

Conclusions

Whether used in combination with quantitative methods in a mixed method design or alone, qualitative methods offer enormous potential to contribute to the field of mental health services research. Although they are distinguished from quantitative methods by features of design (reliance on small samples, purposeful sampling, emphasis on naturalistic inquiry, and iteration between data collection and analysis), data collection (interviews, focus groups, participant observation), and data analysis (grounded theory, phenemonology, holistic perspective), they share with quantitative methods a regard for rigor, validity and reliability. It must be kept in mind, however, that while there are certainly areas of overlap, qualitative methods are not a substitute for quantitative methods, but rather represent a specific set of tools that can be used with greater effectiveness in some phases of the process of scientific inquiry and with less effectiveness in others. Interview or focus group data may indeed be of little value when analyzing the outcomes of a randomized controlled trial; however, such data can enable an investigator to achieve a deeper understanding of process and context of an RCT, develop better instruments for measuring process and outcome, more efficiently target potential study participants, enhance the external validity of the findings, and account for unexplained findings of an analysis of quantitative data.

Acknowledgments

This study was funded through a grant from the National Institute of Mental Health (P30-MH090322: K. Hoagwood, PI).

References Cited

• Aarons GA, Hurlburt M, Fettes D, Willging C, Gunderson L, Chaffin M, Palinkas LA. Collaboration, negotiation and coalescence for interagency-collaborative teams to scale up evidence-based practice. Journal of Clinical Child and Adolescent Psychology in press. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Aarons GA, Wells R, Zagursky K, Fettes DL, Palinkas LA. Implementing evidence-based practice in community mental health agencies: multiple stakeholder perspectives. American Journal of Public Health. 2009; 99 (11):2087–2095. doi: 10.2105/AJPH.2009.161711. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bäärnhielm S, Scarpinati Rosso M, Pattyi L. Culture, context and psychiatric diagnosis: A manual for interviewing according to the Cultural Formulation in DSM-IV. 2009 Retrieved from http://www.mcgill.ca/files/iccc/Culture_context_and_psychiatric_diagnosis_Manual.pdf .
• Beehler GP, Funderbrurk JS, Possemato K, Vair CL. Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: A Delphi study. Implementation Science. 2013; 8 :19. www.implementationscience.com/content/8/1/19 . [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bernard HR. Research methods in anthropology: Quantitative and qualitative approaches. 3rd. Walnut Creek, CA: Alta Mira Press; 2002. [ Google Scholar ]
• Berwick D. The science of improvement. Journal of the American Medical Association. 2008; 299 (10):1182–1184. doi: 10.1001/jama.299.10.1182. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bradley EH, McGraw SA, Curry L, Buckser A, King KL, Kasl SV, Andersen R. Expanding the Andersen Model: the role of psychosocial factors in long-term care. Health Services Research. 2003; 37 :1221–1142. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Brunette MF, Asher D, Whitley R, Lutz WJ, Wieder BL, Jones AM, McHugo GJ. Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. Psychiatric Services. 2008; 59 :989–995. doi: 10.1176/appi.ps.62.4.360. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Byng R, Norman I, Redfern S, Jones R. Exposing the key functions of a complex intervention for share care in mental health: Case study of a process evaluation. BMC Health Services Research. 2008; 8 :274. doi: 10.1186/1472-6963-8-274. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Caudill W. The psychiatric hospital as a small society. Cambridge, MA: Harvard University Press; 1958. [ Google Scholar ]
• Chew-Graham C, Slade M, Montana C, Stewart M, Gask L. A qualitative study of referral to community mental health teams in the UK: Exploring the rhetoric and the reality. BMC Health Services Research. 2007; 7 :117. doi: 10.1.1186/1472-6963-7-117. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cresswell JW, Plano Clark VL. Designing and conducting mixed method research. Thousand Oaks, CA: Sage; 2007. [ Google Scholar ]
• Coffey A, Atkinson P. Making sense of qualitative data. Sage; Thousand Oaks: 1996. [ Google Scholar ]
• Dorsey S, Conover K, Cox JR. Improving foster parent engagement: Using qualitative methods to guide tailoring of evidence-based engagement strategies. Journal of Clinical Child and Adolescent Psychology in press. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Estroff S. Making it crazy: An ethnography of psychiatric clients in an American community. Berkeley, CA: University of California Press; 1981. [ Google Scholar ]
• Gianakis M, Carey TA. An interview study investigating experiences of psychological change without psychotherapy. Psychology and Psychotherapy: Theory, Research and Practice. 2011; 84 :442–457. doi: 10.1111/j.2044-834.2010.02002x. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gilburt H, Slade M, bird V, Oduola S, Craig TKJ. Promoting recovery-oriented practice in mental health services: A quasi-experimental mixed-methods study. BMC Psychiatry. 2013; 13 :167. http://www.boimedcentral.com/1471-244X/13/167 . [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Glaser BG, Strauss AL. The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter; 1967. [ Google Scholar ]
• Goffman I. Asylums: Essays on the social situations of mental patients and other inmates. New York: Doubleday; 1961. [ Google Scholar ]
• Green CA, Polen MR, Castleton DK, Wisdom JP, Vuckovic N, Perrin NA, Paulson RI, Oken SL. Understanding how clinician-patient relationships and Relational continuity of care affect recovery from serious mental illness: STARS study results. Psychiatric Rehabilitation Journal. 2008; 32 (1):9–22. doi: 10.2975/32.1.2008.9.22. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006; 18 (1):59–82. [ Google Scholar ]
• Guba E. CSE Monograph Series in Evaluation 8. Los Angeles: Center for the Study of Evaluation, University of California, Los Angeles; Toward a methodology of naturalistic inquiry in educational evaluation. [ Google Scholar ]
• Harris K, Collinson C, das Nair R. Service-users' experiences of an early intervention in psychosis service: An interpretative phenomenological analysis. Psychology and Psychotherapy: Theory, Research and Practice. 2012; 85 :456–489. doi: 10.1111/j.2044-8341.2011.02043.x. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hohmann AA. A contextual model for clinical mental health effectiveness research. Mental Health Services Research. 1999; 1 :83–91. [ Google Scholar ]
• Hopper K. Qualitative and quantitative research: Two cultures. Psychiatric Services. 2008; 59 :711. doi: 10.1176/appi.ps.59.7.711. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Husserl E. Ideas. New York: Collier; 1962. [ Google Scholar ]
• Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP, Magnabosco J, Rapp CA, Ganju V, Goldman HH. The state policy context of implementation issues for evidence-based practices in mental health. Psychiatric Services. 2007; 58 (7):914–921. doi: 10.1176/appi.ps.58.7.914. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Krueger RA. Moderating focus groups. Thousand Oaks, CA: Sage; 1988. [ Google Scholar ]
• Kuhn T. The structure of scientific revolutions. Chicago: University of Chicago Press; 1970. [ Google Scholar ]
• Lee BR, Munson MR, Ware NC, Ollie MT, Scott LD, Jr, McMillen JC. Experiences of and attitudes toward mental halth services among older youths in foster care. Psychiatric Services. 2006; 57 :487–492. doi: 10.1176/appi.ps.57.4.487. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lynch SE, Cho J, Ogle S, Sellman H, dosReis S. A phenomenological case study of communication between clinicians about attention-deficit/hyperactivity disorder assessment. Clinical Pediatrics. 2014; 53 (1):11–17. doi: 10.1177/0009922813497092. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lyon AR, Ludwig K, Romano E, Koltracht J, Vander Stoep A, McCauley E. Using modular psychotherapy in school mental health: Provider perspectives on intervention-setting fit. Journal of Clinical Child and Adolescent Psychology. 2013 Oct 17; doi: 10.1080/15374416.2013.843460. Epub ahead of print. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Marcus M, Westra H, Angus L, Kertes A. Client experiences of motivational interviewing for generalized anxiety disorder: A qualitative analysis. Psychotherapy Research. 2011; 21 (4):447–461. doi: 10.1080/10503307.2011.578265. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Mendenhall AN, Kapp SA, Rand A, Robbins ML, Stipp K. Theory meets practice: The localization of wraparound services for youth with serious emotional disturbance. Community Mental Health Journal. 2013; 49 :793–804. doi: 10.1007/s10597-013-9646-y. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. 2nd. Thousand Oaks, CA: Sage; 1994. [ Google Scholar ]
• Mittal D, Drummond KL, Blevins D, Curran G, Corrigan P, Sullivan G. Stigma associated with PTSD: Perceptions of treatment seeking combat veterans. Psychiatric Rehabilitation Journal. 2013; 36 :86–92. doi: 10.1037/h0094976. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Morgan DL. Focus groups as qualitative research. Newbury Park, CA: Sage; 1988. [ Google Scholar ]
• Murray LK, Skavenski S, Michalopoulos LM, Bolton PA, Bass JK, Familiar I, Imasiku M, Cohen J. Counselor and client perspectives of trauma-focused cognitive behavioral therapy for children in Zambia: A qualitative study. Journal of Clinical Child and Adolescent Psychology. 2013 Jan 8; doi: 10.1080/15374416.2013.859079. Epub ahead of print. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Padgett DK. Qualitative methods in social work. 2nd. Los Angeles: Sage; 2008. [ Google Scholar ]
• Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research. 2011; 38 (1):44–53. doi: 10.1007/s10488-010-0314-z. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Palinkas LA, Horwitz SM, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in mental health services research: A review. Psychiatric Services. 2011; 62 (3):255–263. doi: 10.1176/appi.ps.62.3.255. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research. 2013 Nov 6; doi: 10.1007/s10488-013-0528-y. epub ahead of print. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Palinkas LA, Schoenwald SK, Hoagwood K, Landsverk J, Chorpita B, Weisz JR the MacArthur Research Network on Youth Mental Health. An ethnographic study of implementation of evidence-based treatments in child mental health: first steps. Psychiatric Services. 2008; 59 (7):738–746. doi: 10.1176/appi.ps.59.7.738. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Patton MQ. Qualitative Evaluation and Research Methods. 3rd. Thousand Oaks, CA: Sage; 2002. [ Google Scholar ]
• Powell BJ, Proctor EK, Glisson CA, Kohl PL, Raghavan R, Brownson RC, Stoner BP, Carpenter CR, Palinkas LA. A mixed methods multiple case study of implementation as usual in children's social service organizations: Study protocol. Implementation Science. 2013; 8 :92. doi: 10.1186/1748-5908-8-92. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Proctor EK, Hascke L, Morrow-Howell N, Shumway M, Snell G. Perceptions about competing psychosocial problems and treatment priorities among older adults with depression. Psychiatric Services. 2008; 59 :670–675. doi: 10.1176/appi.ps.59.6.670. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rappaport FL, Jerzembek GS, Doel MA, Jones A, Cella M, Lloyd KR. Narrating uncertainties about treatment of mental health conditions. Social Psychiatry and Psychiatric Epidemiology. 2010; 45 :371–379. doi: 10.1007/s00127-009-0072-y. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Robins CS, Ware NC, dosReis S, Willging CE, Chung JY, Lewis-Fernández R. Dialogues on mixed methods and mental health services research: anticipating challenges, building solutions. Psychiatric Services. 2008; 59 :727–731. doi: 10.1176/appi.ps.59.7.727. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rodriguez A, Southam-Gerow MA, O'Connor MK. An analysis of stakeholder views on children's mental health services. Journal of Clinical Child and Adolescent Psychology in press. [ PubMed ] [ Google Scholar ]
• Sandu S, Bjerre NV, Dauvrin M, Dias S, Gaddini A, Graecen T, Priebe S. Experiences with treating immigrants: a qualitative study in mental health services across 16 European countries. Social Psychiatry and Psychiatric Epidemiology. 2013; 48 :105–116. [ PubMed ] [ Google Scholar ]
• Scarpinatti Rosso M, Bäärnhielm S. Use of the cultural formulation in Stockholm: A qualitative study of mental illness experience among migrants. Transcultural Psychiatry. 2012; 49 (2):283–301. doi: 10.1177/1363461512442344. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Schutz A. On phenomenology and social relations. Chicago: University of Chicago Press; 1970. [ Google Scholar ]
• Simmons MB, Hetrick SE, Jorm AF. Making decisions about treatment for young people diagnosed with depressive disorders: a qualitative study of clinicians' experiences. BMC Psychiatry. 2013; 13 :335. doi: 10.1186/1471-244X-13-335. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Slade M, Priebe S. Are randomized controlled trials the only gold that glitters? British Journal of Psychiatry. 2001; 179 :286–287. doi: 10.1192/bjp.179.4.286. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Smith JA. Beyond the divide between cognition and discourse: Using interpretative phenomenological analysis in health psychology. Psychology and Health. 1996; 11 :261–271. doi: 10.1080/08870449608400256. [ CrossRef ] [ Google Scholar ]
• Sofaer S. Qualitative methods: What are they and why use them? Health Services Research. 1999; 34 (5, part 2):1101–1118. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Stergiopoulos V, O'Campo P, Gozdnik A, Jeyaratnam J, Corneau S, Sarang A, Hwang SW. Moving from rhetoric to reality: Adapting Housing First for homeless individuals with mental illness from ethno-racial groups. BMC Health Services Research. 2012; 12 :345. doi: 10.1186/1472-6963-12-345. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Strauss AL, Corbin J. Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks, CA: Sage; 1998. [ Google Scholar ]
• Teddlie C, Tashakkori A. Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in the social and behavioral sciences. Thousand Oaks, CA: Sage; 2003. pp. 3–50. [ Google Scholar ]
• Trochim WM. An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning. 1989; 12 :1–16. [ Google Scholar ]
• Turner KM, Sharp D, Folkes L, Chew-Graham C. Women's views and experiences of antidepressants as a treatment for postnatal depression: A qualitative study. Family Practice. 2008; 25 :450–455. doi: 10.1093/fampra/cmn056. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ware NC, Tugenberg T, Dickey B, McHorney CA. An ethnographic study of the meaning of continuity of care in mental health services. Psychiatric Services. 1999; 50 :395–400. [ PubMed ] [ Google Scholar ]
• Watts BV, Shiner B, Zubkoff L, Carpenter-Song E, Ronconi JM, Coldwell CM. Implementation of evidence-based psychotherapies for posttraumatic stress disorder in VA specialty clinics. Psychiatric Services, in Advance. 2014 Jan 15; doi: 10.1176/appi.ps.201300176. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Woltman EM, Whitley R, McHugo GJ, Brunette M, Torrey WC, Coots L, Lynde D, Drake RE. The role of staff turnover in the implementation of evidence-based practices in health care. Psychiatric Services. 2008; 59 :732–737. doi: 10.1176/appi.ps.59.7.732. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Yin RK. Case study research design and methods. 3rd. Thousand Oaks, CA: Sage; 2003. [ Google Scholar ]