11 Focus Groups

Abstract

Focus groups are a qualitative method which consists of a researcher leading a collective conversation with a group of (commonly 4 to 8) people, guided by questions for them to comment on. Focus groups can be a way of involving users and diverse stakeholders’ viewpoints and experiences in the evaluation of a given intervention. They are suitable at different stages of the policy process and for different evaluation approaches, often in combination with other qualitative and/ or quantitative methods.

Keywords: Qualitative methods, focus groups, group interviews, participatory evaluation

I. What does this method consist of?

Focus groups consist of one or more conversations with a group of people assembled for discussion. Focus groups are led by a researcher (and often include an observer), aiming to gain knowledge of different possible outcomes in areas such as selling (marketing), influencing decisions (politics, health behaviour), or assessing the worth of public interventions and policies (monitoring and evaluation). Trained researchers facilitate discussions with a set of unstructured and/or structured questions for the group to comment on. These conversations can be stimulated by prompts such as photos, videos, vignettes, games, etc and by decision-making techniques such as informal voting methods. Focus groups can be conducted in a variety of formats (digital, analogue, virtual) and with participants of similar characteristics of interest (referred to as “homogeneous focus groups”, such as a focus group comprising of teachers) or diverse characteristics (referred to as “heterogeneous focus groups”, such as a focus group comprising of teachers, students and parents).

Focus groups are a primary qualitative research method, belonging to the family of group-based discussion methods. Similar to other qualitative data, there is no agreement in the research methods literature on the optimum number of participants for a focus group or the suitable number of groups. At design stages, it is useful to present focus group sizes in ranges because there are many contingencies that can impact the number of participants attending groups. Some authors favour smaller groups (n=3-5) because they have greater potential to explore complex topics in depth. For example, richer information can be obtained by conducting two groups of four participants than one group of eight participants. Some recommend medium size groups (n=6-8), while others suggest bigger groups (n=6-12) to capture a greater variety of views. The duration of the discussion depends on group size and topic but, as a rule of thumb, 90 minutes are necessary for all discussants to have the chance to express their views. Durations longer than two hours can increase participant burden and will also increase the risk of deterring people from attending in the first place.

Many expressions are used in evaluation to describe group data collection methods and there are geographic preferences between the use of “focus group” vs “group interview.” A key distinction is that focus groups highlight the significant role of group dialectical processes (e.g. norms, dynamics, non-verbal communication) that can assist evaluators in gaining knowledge about group views and subgroup agreements and disagreements. Conceptual differences between many of the group data collection terms are often unclear and there is not always consensus on how they are different from each other. Often group/stakeholder sessions (community group meetings, advisory groups, public engagement workshop consultation events, knowledge cafes, expert meetings, collective facilitated conversation with groups, etc) are not designed or conducted as per standard focus groups discussion format. Although sometimes these sessions can be directed at particular policy beneficiaries’ groups, all those require less preparatory work, no structured facilitation from researchers, and lack post-event content/transcript formal analysis.

II. How is this method useful for policy evaluation?

Although policy evaluation has been dominated by the search for hard facts through experimental and quantitative approaches, policy makers also have a preference for user/customer involvement, and focus groups have the potential to support this participatory aim. Alongside in-depth interviews, focus groups are one of the most used qualitative social research methods in policy evaluation. The distinctive features of focus groups are attractive to policy makers, such as exploring contrasting meanings, values, experiences, viewpoints and behaviours from different subgroups of stakeholders, and capturing the complexity of policy implementation contexts and processes. The political value of focus groups is often as important as the specific information about values and multiple viewpoints that the groups can provide.

On their own or combined with other research methods, focus groups are used in many evaluation approaches (e.g. theory-driven and theory-based, process and outcome/impact evaluations, developmental, participatory and empowerment evaluation), for a range of purposes, and at different stages of the policy process (planning, implementation, monitoring, assessment, successive programming cycles). They are suitable for ex ante and ex post evaluation approaches, and are often used in evaluability assessments, needs assessments, programme theory development, instrument and survey development, implementation, utilisation-focused and formative evaluations.

Focus groups are mostly useful to answer exploratory evaluative questions (why and how) because they provide a dynamic means to portray policies in action. They have the potential to increase understanding of:

• a problem, or a policy to approach a problem, and how it is perceived and experienced by different stakeholders (users, front-line staff, management), their expectations and solutions proposed by them.

• to get feedback on quality, use and satisfaction related to the activities and resources delivered by the policy. What worked well, for whom, and what did not work as intended, why and in what circumstances this happened.

• the policy implementation process (e.g. the management, the partnerships with other institutions/departments, the delivery of policy activities and resources).

• to discern the types of changes assumed/expected (theories of change) and produced (if any) from different user perspectives and in different policy contexts across time and space.

• to explore evaluation indicators/criteria when they are not clear or alternative criteria are sought.

• to understand people’s experiences of outputs and short-medium-long term outcome patterns (intended and unintended) observed in different macro-meso-micro policy contexts and/or as a consequence of the changes (activities and resources) brought about by the policy.

• to develop and pre-test other qualitative and quantitative data collection instruments such as interviews, experiments and surveys.

III. Two examples of the use of this method: developing indicators and assessing the implementation of a childhood development programme

Focus groups should be used in policy evaluation according to the type of evidence to be generated. For example, focus groups – combined with other primary and secondary methods – are often used to develop evaluation indicators (e.g. participation rates, incidence) that can help answer evaluation questions by marking accomplishments (outputs/outcomes) in a specific and measurable way. Involving beneficiaries and other stakeholders to develop indicators could make the policy relevant to them and enhance buy-in of evaluation findings. EVALSED (European Commission, 2008), a resource providing guidance for the evaluation of socio-economic development policies in the European Union, gives an example of using focus groups with policy beneficiaries (e.g. representatives of regional enterprises) to develop evaluation indicators in an economic development policy in Benton Harbour (Michigan, USA).

Formative evaluations, which aim to develop policies by examining their implementation, often use focus groups. The formative evaluation of the UNICEF Early Childhood Development (ECD) Project of the Integrated Maternal and Child Health and Development Programme (2017-2020) in China (Zhou Hong et al. 2022), employed a theory-based, utilisation-focused, mixed-methods design that included focus groups. The evaluation results provided evidence to advocate for the national scale-up of the ECD model and informed the design of the National Health Commission-UNICEF Scaling up of Early Childhood Development Program 2021-2025. Focus group discussions with younger parents/caregivers identified needs for nurturing care skills and this evidence was a driving force for recommending the scale-up of the ECD. Stigma in home visits was also raised in some group discussions and additional attention to privacy protection was recommended for scaling-up. Focus groups with administrators reinforced recommendations to increase funding for the implementation of three types of services, guarantee service frequency and increase service coverage.

IV. What are the criteria for judging the quality of the mobilisation of this method?

Since there are multiple evaluation approaches that differ greatly in their philosophical and methodological premises, a single set of quality indicators for conducting focus groups in evaluation does not exist. This is because each of those approaches has diverse and contradictory assumptions and what matters in terms of ‘quality’ varies according to these assumptions.

Similarly, qualitative research is not a uniform approach, comprising many different qualitative traditions based on different paradigms, with diverse philosophical assumptions, that a single quality framework could not address. The area of “qualitative data quality criteria” is controversial, with various positions and many classificatory suggestions available, which range from a total rejection of the notion of criteria, to those who propose similar criterion for quantitative and qualitative research.

Consequently, although there are abundant quality criteria on when to use, how to design, recruit, conduct and analyse focus groups, there are no agreed standards for judging quality in qualitative research evaluations. Focus groups come in many formats, and this is why practical, design and quality issues can take on rather contrasting characters. For example, choice of venue is important for “real life” focus groups (as opposed to virtual reality focus groups) since successful recruitment may depend on venue accessibility and practicalities (travel costs, refreshments, audio recording); issues of duration and facilitation are always important, but they will be taken to another level in computer-mediated discussion forms (synchronous or asynchronous).

Spencer et al. (2003, 16) proposed a general framework to support quality indicators on four qualitative methods, including focus groups. This framework is based on four essential guiding principles: 1) To be contributory in advancing wider knowledge or understanding; 2) To have a defensible design and strategy that aims to answer the given evaluation questions; 3) To be rigorous through systematic, transparent data collection, and analysis and interpretation of data; 4) to be credible by offering justifiable, defensible and plausible arguments about the significance of the data generated.

Ryan et al. (2014) proposed that evaluators consider core questions to maximise their learning when conducting focus groups and to improve the credibility of focus group evidence, such as: “Did the focus group participants establish common ground in conversation or primarily act as individuals?”; “What were the power dynamics between the moderator and participants, both as a group and as individuals?”; “What were the relations among the participants – collective or dominant?”

V. What are the strengths and limitations of this method compared to others?

The strengths of using focus groups include:

• In a group, people can build upon/challenge one another’s responses and think of ideas that they may not have thought of on their own. This rich blend of perspectives and disagreements can enlighten researchers on policy complexities, often not attainable from less dynamic methods.

• The flexible format is conducive to exploration of unanticipated outcomes and contextual differences.

• Focus groups are often recommended as a time saving and cost-efficient method but the evidence for such assertions is unclear.

Focus groups present the following limitations:

• They are better used in a mix of method designs instead of as a stand-alone method.

• They are not suitable for the discussion of overly sensitive and/or controversial topics because people are less likely to open up about those in a group and the promise of confidentiality and anonymity is compromised.

• They often do not provide a high level of nuance or detail.

• Focus groups can be challenging for people with physical and communication access needs. Inclusion strategies for all abilities are needed, such as choosing accessible locations and rooms, conducting discussions online, smaller sample size focus groups, etc.

• Culturally responsive focus groups must be mindful not only of language and cultural identities but in some cultures, they may be better replaced with other group conversational decolonised methods, such as sharing circles based on open-structured storytelling.

• Focus groups have unpredictable composition and dynamics. Some groups of stakeholders are notably hard to recruit for group discussions.

• Those who are comfortable speaking in front of a group are more likely to be recruited.

• Discussions can be side-tracked and/or dominated by vocal individuals/group opinion leaders. Status differences between researchers and participants, or among participants, will influence discussions.

Some bibliographical references to go further

Cohen Miller, Anna. and Durrani, Naureen. and Kataeva, Zumrad. and Makhmetova, Zhadyra. 2022. “Conducting Focus Groups in Multicultural Educational Contexts: Lessons Learned and Methodological Insights.” International Journal of Qualitative Methods, 21: 1–10. https://op.europa.eu/en/publication-detail/-/publication/752dd092-3ea5-429c-96dc-4507d0cda886.

European Commission. 2008. “EVALSED: The Resource for the Evaluation of Socio-Economic Development.” https://op.europa.eu/en/publication-detail/-/publication/752dd092-3ea5-429c-96dc-4507d0cda886.

Krueger, Richard. and Casey, Mary Anne. 2000. Focus Group: A Practical Guide for Applied Research. Thousand Oaks, California: Sage.

Manzano, Ana. 2022. Conducting focus groups in realist evaluation. Evaluation, 28(4): 406-425.

Ryan, Katherine. and Gandha, Tyszha. and Culbertson, Michael. and Carlson, Crystal. 2014. “Focus Group Evidence: Implications for Design and Analysis.” American Journal of Evaluation, 35(3): 328–45.

Spencer, Liz. and Ritchie, Jane. and Lewis, Jane. and Dillon, Lucy. 2003. “Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence.” London. https://www.cebma.org/wp-content/uploads/Spencer-Quality-in-qualitative-evaluation.pdf.

Zhou, Hong. and Yang, Li. and Wang, Yan. and Liu, Shuang. and He, Hong. 2022. “Formative Evaluation of the National Health Commission-UNICEF Early Childhood Development Project of the Integrated Maternal and Child Health and Development Programme (2017-2020).” https://www.unicef.org/evaluation/reports#/detail/13964/formative-evaluation-of-the-national-health-commission-unicef-early-childhood-development-project-of-the-integrated-maternal-and-child-health-and-development-programme-2017-2020.