17 Mixed methods systematic reviews

Abstract

Mixed methods systematic literature reviews, also named mixed studies reviews, consist of reviewing available work (including evaluations already carried out) on a given topic by incorporating studies using qualitative, quantitative and mixed methods. This type of literature review allows for a better understanding of complex interventions and phenomena. By encompassing a diversity of questions, they are particularly useful for informing public decision-making.

Keywords: Mixed methods, qualitative studies, quantitative studies, systematic review, literature review

I. What does this method consist of?

The literature review process consists of summarising, combining, analysing, commenting on or critiquing the literature on a given topic. Mixed methods systematic literature reviews, also named mixed studies reviews, are unique in that they include a variety of types of studies to better understand complex phenomena: quantitative studies (e.g. studies that measure the effects of an intervention or the magnitude of a problem), qualitative studies (e.g. studies that focus on people’s experiences), and mixed methods studies (i.e. studies that use both quantitative and qualitative methods).

Mixed methods systematic reviews are part of the larger family of systematic literature reviews, i.e. a type of literature review that aims to answer a research question by following a pre-defined approach to the identification, selection, appraisal and synthesis of relevant studies. It is considered to be one of the most rigorous types of review since it minimises errors and biases that may occur during the review process. These reviews therefore use explicit methods and report them transparently so that they can be replicated. In addition to the systematic review, there are a variety of types of reviews that use a systematic approach, such as scoping review, rapid review and overview of reviews.

In general, conducting a mixed methods systematic review follows eight steps:

1. Formulate a research question(s) – the formulation of questions can be guided by a cursory exploration of the existing literature on the topic of interest;

2. Define eligibility criteria (inclusion and exclusion) for the selection of articles;

3. Identify literature sources to ensure a comprehensive search such as searching bibliographic databases (e.g. PubMed, Health Policy Reference Center, International Political Science Abstracts, Europa World, Worldwide Political Sciences Abstracts, Web of Science, JSTOR, SocINDEX), consulting the table of contents of scientific journals on the topic of interest, searching websites on the topic, consulting the reference list of selected articles or articles that have been cited, and contacting experts;

4. Develop a literature search strategy with the assistance of a specialised librarian who is familiar with literature search techniques in databases and other sources;

5. Selecting relevant documents in two stages: selection based on document titles and abstracts, and selection based on full-text articles;

6. Assessing the quality of the selected documents using critical appraisal tools;

7. Extracting data from the selected documents using a form that specifies all the data to be extracted; and

8. Synthesising the extracted data, i.e. analysing the data extracted in the review into a coherent whole to answer the research question(s). In mixed methods systematic reviews, the synthesis should also consider how the qualitative and quantitative data will be integrated. In general, two main synthesis designs can be used to synthesise qualitative and quantitative data: (a) convergent synthesis designs in which data are synthesised simultaneously (e.g. quantitative and qualitative studies are synthesised separately with different synthesis methods and then the results of the two syntheses are compared) and (b) sequential synthesis designs that involve at least two dependent phases of synthesis (e.g. qualitative studies are synthesised first and then the results of this synthesis inform the synthesis of the quantitative studies).

For more information on each of these steps and on the synthesis designs, you can consult the references at the end of this document as well as this website: http://toolkit4mixedstudiesreviews.pbworks.com.

II. How is this method useful for policy evaluation?

Mixed methods systematic reviews are relevant for policy evaluation, as they allow for a deeper understanding of complex phenomena and interventions. Complex phenomena are often characterised by a multiplicity of actors involved, and a diversity of intervention models and factors influencing their success. Various types of studies can be used to assess these complex phenomena. Thus, the synthesis of these complementary studies will provide a more comprehensive understanding of the state of knowledge on a complex phenomenon.

The mixed methods systematic review provides a broader and more complete picture of the literature on a given topic. It also allows for the combination of complementary questions such as: How effective is a policy? Why is the policy effective or not? How does the policy work? What factors hinder or facilitate the implementation of the policy? To answer these questions, it is necessary to include quantitative studies to answer questions about policy effectiveness and qualitative studies that address the why and how questions. The answers to these complementary questions can lead to better decision-making by policy makers, managers and practitioners.

Other advantages of mixed methods systematic reviews were highlighted such as allowing for a better understanding of the results obtained in quantitative studies from qualitative studies (or vice versa), considering a diversity of perspectives (e.g. perspectives of decision makers and users), corroborating findings obtained from different evidence, and enhancing the credibility and validity of conclusions.

III. An example of the use of this method: the fight against smoking among young people

A mixed methods systematic review was conducted by researchers at the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) in the UK to inform the development of policies to reduce youth smoking rates (Sutcliffe, Twamley, Hinds et al., 2011). In this review, they addressed three main research questions: a) What are the most common sources in which young people aged 11-18 years access retail and non-retail tobacco products and do the sources vary by factors such as age and gender? (b) What are young people’s perceptions of access to tobacco products and what do they believe are the barriers and facilitators to accessing tobacco products; and (c) What types of interventions to limit access to tobacco products by young people in non-retail settings have been evaluated and how do these interventions address the barriers and facilitators identified as important by young people in the UK?

For this mixed methods systematic review, a literature search of over 100 sources of information was conducted (e.g. databases, websites, citation searches). This search identified six qualitative studies, seven surveys, and sixteen intervention studies. Two people were independently involved in the selection of studies, quality assessment, data extraction, and synthesis. The syntheses for each type of study included (surveys, qualitative studies, intervention studies) were carried out separately. For integration, the results of these syntheses were then compared in two ways: a) to assess the level of concordance between the results of the surveys and qualitative studies regarding young people’s sources of tobacco products and their access patterns by gender, age and smoking status (occasional or regular); and b) to assess the extent to which the interventions addressed the barriers and facilitators identified as important by young people in the qualitative studies.

In light of the findings of this review, three main implications for the development of policies to reduce youth smoking rates were formulated. Firstly, the studies found that tobacco products were easily accessed through friends and peers in schools (social access). Young people described this social access as ubiquitous, organised, and visible. Thus, the development of stricter regulations in schools to reduce social access should be explored. Secondly, the results of the studies indicate that the implementation of retail regulation was variable. It is therefore necessary to explore the reasons for uneven implementation and to identify ways to combat lax implementation of regulation in smaller retail stores. Thirdly, the findings of this review suggest the need to address adult complicity to purchase tobacco products such as by family, friends and strangers (proxy purchase).

In this mixed methods systematic review, the use of data from different types of studies allowed for the identification of different modes of access used, a better understanding of young people’s experiences and views on access to tobacco products, and the exploration of potential avenues for intervention. Also, the mixed methods nature of the review, which combines survey data, research on the views of young people in the UK and interventions addressing non-retail access to tobacco products, has provided contextualised evidence for policy development.

IV. What are the criteria for judging the quality of the mobilisation of this method?

To judge the quality of systematic reviews, it is important to understand the potential sources of errors and biases that can influence the results obtained. Four biases are presented in this paper: identification bias, reporting bias, selection bias and interpretation bias.

Identification bias occurs when relevant studies on the topic of interest are not identified. This bias is related to the literature search and the sources of identification. In order to conduct a comprehensive search for all studies relevant to the research question, it is important to diversify literature sources, use different databases and develop rigorous literature search strategies with the collaboration of specialised librarians.

Reporting bias, of which publication bias is the best known, occurs when the nature, direction or strength of a study’s results influence its publication. For example, it has been shown that studies with positive effects are more likely to be published and published more quickly in scientific journals than those with negative results. This may lead to an over-representation of studies showing positive effects and may affect the conclusions of the systematic review. To minimise this bias, it is recommended to diversify the sources of data and the types of documents to be included, such as scientific reports from research centres, and master’s theses and doctoral dissertations.

Selection bias occurs when the selection of studies is arbitrary or influenced by particular motivations or beliefs. For example, a researcher may believe that an intervention is important and decide to include only studies that have shown that the intervention is effective. This will bias the results of the review. To minimise this bias, it is important to define clear eligibility criteria prior to selection and to involve two people in the selection process.

Interpretation bias is related to the persons’ misinterpretation of the studies. This bias can be minimised by involving at least two people in data extraction, quality assessment and data synthesis. Furthermore, in a mixed methods systematic review, it is recommended to have a team with complementary expertise in qualitative and quantitative research to facilitate the synthesis and judgement of studies’ quality.

V. What are the strengths and limitations of this method compared to others?

A mixed methods systematic review makes it possible to answer a variety of questions, to take advantage of the complementarity of quantitative and qualitative data, and to gain a thorough and complete understanding of a complex phenomenon. Also, using a rigorous and explicit methodology helps to minimise potential errors and biases that could influence the validity of a review’s findings. However, various challenges can arise in its operationalisation.

One important challenge is the time and resources required. The duration of a systematic review can vary from 6 to 24 months. Various factors can influence its duration such as the research questions to be addressed, the number of people involved, the number of documents to be analysed, and the synthesis method(s) to be used. Also, including a variety of study types in a mixed methods systematic review increases the volume of material to be identified, screened, extracted and analysed. It is therefore important to ensure that resources are available and that the choice to conduct a mixed methods systematic review is well justified.

The questions that can be studied in a systematic review depend on the available literature. For example, in the context of a mixed methods systematic review, a research team might be interested in identifying studies on the effects of an intervention and others on the users’ perspective on the intervention. However, let us imagine that the literature search only identifies studies on the effects and none on the user perspective of the intervention of interest. In this example, the review is not mixed since only one type of study is synthesised. In order to guide the specific research questions that could be addressed on a topic of interest in a mixed methods systematic review, it may be useful to conduct a preliminary, cursory exploration of the existing literature.

Another challenge is the integration of data, i.e. how the qualitative and quantitative components are combined. This is a key feature of the mixed methods systematic review, which allows the full range of results from the various types of studies selected to be integrated in order to provide a deeper understanding of the topic of interest and to make recommendations that reflect the full body of literature covered. A review that does not have integration could be considered to include several independent reviews rather than a mixed methods review. It is therefore essential that the way in which the data is integrated is well described and that the added value of this integration and its limitations are well reflected.

Some bibliographical references to go further

Granikov, Vera. and Hong, Quan Nha. and Pluye, Pierre. 2022. “Mixing Qualitative and Quantitative Evidence in a Systematic Review: Methodological Guidance with a Worked Example of Collaborative Information Monitoring.” In Handbook of Research on Mixed Methods Research in Information Science, edited by Ngulube, Patrick. 125-146.

Heyvaert, Mieke. and Hannes, Karin. and Onghena, Patrick. 2016. Using Mixed Methods Research Synthesis For Literature Reviews: The Mixed Methods Research Synthesis Approach. Thousand Oaks, CA: SAGE Publications.

Hong, Quan Nha. and Pluye, Pierre. and Bujold, Mathieu. and Wassef, Maggy. 2017. “Convergent and sequential synthesis designs: Implications for conducting and reporting systematic reviews of qualitative and quantitative evidence.” Systematic Reviews, 6(61): 1-14. https://doi.org/10.1186/s13643-017-0454-2.

Hong, Quan Nha. and Rees, Rebecca. and Sutcliffe, Katy. and Thomas, James. 2020. “Variations of mixed methods reviews approaches: A case study.” Research Synthesis Methods, 11 (6): 795-811. https://doi.org/10.1002/jrsm.1437.

Lizarondo, Lucylynn. and Stern, Cindy. and Carrier, Judith. and Godfrey, Christina. and Rieger, Kendra. and Salmond, Susan. and Apostolo, Joao. and Kirkpatrick, Pamela. and Loveday, Heather. 2020. “Chapter 8: Mixed methods systematic reviews.” In JBI Manual for Evidence Synthesis, ed. by E. Aromataris and Z. Munn. Adelaide: JBI https://jbi-global-wiki.refined.site/space/MANUAL.

Sutcliffe, Katy. and Twamley, Katherine. and Hinds, Kate. and O’Mara, Alison. and Thomas, James. and Brunton, Ginny. 2011. Young people’s access to tobacco: A mixed-method systematic review. EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London (London). https://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3301.