6 Experimentation in the Laboratory

Abstract

Laboratory experimentation makes it possible to directly measure the attitudes and behaviour of individuals and to evaluate the causal effect of a variable on these attitudes and behaviour. To do this, individuals are put in a situation where they are asked to perform a certain number of tasks for which as many elements as possible are controlled (such as the duration of the task and the type of information given to participants). This approach can help to anticipate ex ante how individuals will respond to an intervention or can be used ex post to measure changes in behaviour following an intervention. It is particularly useful for uncovering non-conscious behavioural biases.

Keywords: Quantitative methods, within-/between-participant method, laboratory experimentation, causal effect, behaviours, attitudes, non-conscious behavioural bias, internal/external validity, automatic/non-automatic response

I. What does this method consist of?

In a simple way, in a laboratory experiment, participants perform a given task, designed to measure their behaviour. The first step in laboratory experimentation is therefore to establish an experimental protocol for measuring the individual’s behaviour. Classically, these experiments rely on a computer task, which will make it possible to measure not only the participants’ choices but also other data that may prove particularly informative, such as response time. These tasks can be aimed at measuring participants’ preferences and perceptions as well as the way they learn or reason. Thus, laboratory experiments are particularly used by fields that are directly concerned with people’s behaviours and perceptions, such as cognitive science and psychology, including social, developmental and political psychology, as well as economics and educational science. Most of these protocols are based on measuring participants’ choices between different options or their evaluations of these options on a scale. For this purpose, different types of material (or stimuli) can be presented to the participants (images, texts, videos, sounds etc.). Thus, this method makes it possible to measure attitudes and behaviours directly, which can be particularly useful when it comes to behaviours or attitudes that participants tend not to report or of which they are not aware, even though these attitudes may have a significant influence on their behaviours, as is the case for implicit gender bias.

In addition to offering the possibility of directly measuring behaviour, laboratory experiments also make it possible to measure how behaviour can be influenced by a specific context. This is the core of the scientific experimental method: by comparing participants’ behaviour in different conditions, one in which the factor of interest (the one whose influence is being studied) is present and one in which it is absent, it is possible to assess the causal link between this factor and the behaviour being studied. However, as these studies are conducted in laboratory settings, this factor of interest must be extracted from the real context to be studied experimentally. For example, when studying the acceptability of a new drug, its price, efficacy and side effects can be studied together or separately using fictitious choices in order to estimate their influence on the participants’ perceptions. Thus, laboratory experiments require a thorough analysis of the factors that may affect the behaviour of interest. This notion of comparison extends beyond the choices themselves and can also be applied to different contexts or conditions. For example, comparing a condition in which participants have access to information on the percentage of female students in each secondary school stream with a condition in which this information is not given allows one to estimate the effect of this type of information on students’ orientation choices.

These comparisons can be made by presenting all contexts or choices to each participant or by presenting only one type of context or choice to each participant. The first method, called within-participant, allows an accurate estimation of these effects by ruling out the possibility that the observed differences are due to factors other than those manipulated in the experiment (such as demographic factors). On the other hand, the second method, called between-participant, does not completely rule out the existence of non-measured explanatory variables, but is necessary when the two manipulated conditions are incompatible. For example, once participants have received information on the percentage of female students in each stream, their choices will most likely be influenced by this factor even if this information is no longer available.

The implementation and use of laboratory experiments therefore require several stages of theoretical reflection, requiring both an understanding of this method and a detailed analysis of public policies, in order to guarantee the quality of the data collected (the internal validity of the experiment) and their capacity to explain behaviours and situations relevant to public policies (the external validity of the experiment).

II. How is this method useful for policy evaluation?

The laboratory experimentation method has a dual purpose for policy evaluation. Firstly, it offers a new tool for measuring the target behaviours of policies (the behaviours that the policies seek to modify), offering complementary measures to existing tools such as questionnaires. It can therefore be integrated into the panel of tools that can be mobilised ex post to measure changes in behaviour following the implementation of an intervention or a public policy.

It also allows for a better understanding of the behaviour of interest, to evaluate its key components and to inform the development of public policies. It thus empirically enriches ex ante knowledge of target behaviours to enable the development of better adapted and thus potentially more effective public policies.

III. Examples of the use of this method for policy evaluation in the fields of education, anti-discrimination and urban cleanliness

Laboratory methods have been used in the field of education to evaluate the effectiveness of different interventions, such as sports, meditation or drama, on the executive functions of children and adolescents. Executive functions are a concept from cognitive science that encompasses the psychological processes involved in performing goal-directed actions, requiring, among other things, the use of action planning, inhibition of competing behaviours, and the smooth transition from one action to another. They have been shown to be associated with several measures of school, academic and occupational success, leading to the development of interventions specifically aimed at improving them in children and adolescents. As executive functions are robustly measured by laboratory experiments, such as tasks in which participants must inhibit an automatic response in order to provide a non-automatic response, laboratory experiments have been used to assess the effectiveness of these interventions. For example, to assess the effectiveness of a four-week meditation programme for 9-11 year old students, Parker and colleagues used a Flanker task, a well-known executive function task, to compare the correct response rates of participants in different conditions: when they were asked to indicate the orientation of a target image surrounded by other similar images and when they were asked to indicate only the orientation of these other images (Parker et al., 2014). While this example illustrates how cognitive science concepts and the associated methods can be mobilised for public policy evaluation, it is important to note that these methods can be combined with tools from other fields such as questionnaires. For example, several interventions aiming at reducing racist bias have combined measures of explicit racism, obtained through questionnaires, and of implicit racism, measured through laboratory experiments, in order to get as complete a picture as possible of the effects of these interventions. An example of this is the study published by Devine et al. in 2012, which these researchers found that an intervention combining an explanation of the existence of implicit racist biases and the presentation of strategies to reduce these biases that was conducted on American students did not have a significant effect on implicit biases but did lead to a reduction in racist biases over a two-month period (Devine et al., 2012).

Laboratory experimentation methods have also been applied to assess ex ante the possible effects of new policies. For example, drawing on the policy literature on the importance of bin visibility in reducing street litter, Abdel Sater and colleagues evaluated the potential effectiveness of an intervention to change the colour of street bin bags in a laboratory setting. To do this, they compared the ability of participants to detect bins in street photos based on the colour of the garbage bags. The colour of the bags was manipulated by computer from real photos, so that the experimental task was as close as possible to real conditions, but also as controlled as possible: only the colour of the bags differed between the photos with the grey bags and those with the red bags. This study demonstrated the potential effectiveness of this simple, low-cost intervention on bin visibility (Abdel Sater et al., 2020). Although this example has not yet been translated into the implementation of a real intervention, it illustrates how laboratory experiments can be integrated into the public policy cycle.

IV. What are the criteria for judging the quality of the mobilisation of this method?

Whether its use is ex post or ex ante, the first element to consider in assessing the relevance of using laboratory experimentation for policy evaluation is the alignment between the behaviour of interest, that which is directly related to the policy question, and the behaviour measured in the laboratory. This idea is fundamental for laboratory experiments to be truly useful for policy evaluation and not just a marketing tool. More precisely, laboratory experiments sometimes use abstract tasks, often initially designed to assess fundamental psychological mechanisms such as motivation. It is therefore necessary to ensure that the behaviour measured experimentally is robustly associated with the behaviour of interest as observed in real-life situations. This question is all the more important as laboratory experiments make it possible to measure not only explicit attitudes, those that participants are prepared to report in interviews or surveys, but also implicit attitudes, of which the participants themselves are not necessarily aware. While the latter type of attitude is of great theoretical interest, it is only weakly predictive of people’s behaviour in everyday life and only predicts behaviour in specific situations, such as when people have to make a decision extremely quickly. Thus, an intervention may not have a significant effect on implicit attitudes but still change participants’ behaviour. Both levels of measurement can be useful for in-depth policy evaluation and anticipating potential unpredicted effects but using only an implicit level of measurement for policy evaluation can lead to misinterpretations of policy effectiveness.

On the other hand, it is important, as with any tool used in the framework of public policy evaluation, to consider the size of the effects obtained. Indeed, the artificial context in which effects are observed in laboratory experiments calls for caution when mobilising these results for the evaluation of public policies. These often highly artificial conditions and tasks, although they make it possible to isolate the behaviour and factors of interest as much as possible, can also lead to biased interpretations when it comes to generalising these results to real situations. Indeed, an experiment in which only one type of information is given (for example, the name of the newspaper in which an article was published), can lead to an overestimation of the weight of this type of information in the decisions of individuals, because unlike the experimental context, in a real context individuals can base their choices on a multitude of information. The mobilisation of laboratory experimentation for the evaluation of public policies therefore requires taking into account the experimental protocol used as a whole, i.e., not only the type of choice that was measured, but also the type of information to which the participants had access.

Finally, in the case of ex ante use, it is also important to consider the population on which the results were obtained in order to assess whether these results can be used for the target population of the public policy. Indeed, behavioural results obtained only on a particular population may not be valid in another population. These differences between populations are notably important to take into account when the analysis specifically aims at comparing different populations and when the experimental protocol is used in a different population from the one on which it was initially tested. In both of these cases, it is necessary to consider that variations in behaviour observed experimentally may be due to the structure of the experimental design itself and not to differences in the behaviour of interest. For example, differences in the participants’ level of concentration on the experimental task may generate differences in behaviour that do not reflect real differences in the target behaviour. It is therefore crucial that the type of experiment chosen be consistent with the target population(s) so as not to artificially create differences in behaviour between populations or to underestimate or overestimate the existence of certain behaviours in these populations.

Finally, in addition to these elements directly linked to the mobilisation of laboratory experiments for the evaluation of public policies, there are general criteria for evaluating the quality of laboratory experiments. These criteria are based in particular on the evaluation of the sensitivity of the experiment and its results to the influence of behavioural bias and randomness. To this end, the use of specific types of formulation, the repetition of each question, the use of a variety of experimental material controlled on key elements (such as the use of a series of different but similarly expressive women’s and men’s faces to assess gender bias) and the randomisation of the presentation of the different elements of the experiment (the order of presentation of questions and conditions for example) are classically implemented to ensure the reliability of the results of experiments conducted in a laboratory.

V. What are the strengths and limitations of this method compared to others?

The two main advantages of the experimental laboratory method are, on the one hand, that it makes it possible to test the existence of causal links between a factor or context and a behaviour and, on the other hand, that it offers a specific tool for measuring behaviour and attitudes. However, it is important to note that the criteria necessary to conduct a reliable laboratory experiment make this method sometimes more restrictive than other methods. For example, laboratory experiments are often longer than questionnaire surveys, making this method more expensive. At the same time, the need to control for a large number of factors limits the exploratory nature of this method and makes it more appropriate for measuring a specific behaviour or evaluating a given hypothesis.

Furthermore, the highly controlled context of laboratory experiments limits the possibility of directly interpreting the results of these experiments in terms of behaviour outside the laboratory. Indeed, the behaviours of interest are sometimes better predicted by explicit responses than by measurements made during laboratory experiments. However, laboratory experiments make it possible to measure behaviours that are difficult or impossible to identify in interviews or to measure in traditional surveys. Indeed, they offer the possibility of measuring implicit behaviours and are less sensitive to the recurrent biases observed with other methods, in particular the social desirability bias, i.e., the desire of participants to show themselves in the best light and to respond according to what they perceive to be a social norm, although this remains a risk in laboratory experiments. Thus, laboratory experiments hold particular promise for assessing the effectiveness of public policies in changing not only the behaviour of individuals but also implicit biases that can have important long-term effects.

Cited references

Abdel Sater, Rita. and Mus, Mathilde. and Wyart, Valentin. and Chevallier, Coralie. 2020. A zero-cost attention-based approach to promote cleaner streets.

Devine, Patricia. and Forscher, Patrick. and Austin, Anthony. and Cox, William. 2012. Long-term reduction in implicit race bias: A prejudice habit-breaking intervention. Journal of experimental social psychology, 48(6): 1267-1278.

Parker, Alison. and Kupersmidt, Janis. and Mathis, Erin. and Scull, Tracy. and Sims, Calvin. 2014. The impact of mindfulness education on elementary school students: Evaluation of the Master Mind program. Advances in School Mental Health Promotion, 7(3): 184-204.

Some bibliographical references to go further

Bordens, Kenneth. and Abbott, Bruce. 2014. Research Design and Methods: A Process Approach. McGraw Hill.

Gawronski, Bertram. 2009. Ten frequently asked questions about implicit measures and their frequently supposed, but not entirely correct answers. Canadian Psychology/Psychologie canadienne, 50(3): 141-150.

Reis, Harry. and Judd, Charles. 2000. Handbook of research methods in social and personality psychology. Cambridge University Press.