9 Direct Observation and Ethnography

Abstract

Direct observation or ethnography is a qualitative method that consists in directly observing the social situation under study – for example, the implementation of a public policy – implying a physical presence of the researcher in the situation at hand. It is a demanding method in terms of the commitment it requires (long-term physical presence in the field, systematic note-taking). It is particularly useful to account for the reality of practices and interactions, at a distance from official discourse.

Keywords: Qualitative methods, ethnography, direct observation, policy implementation, semi-structured interview, interactions, case study

I. What does this method consist of?

Direct observation derives from the practice of ethnographic observation, which is an old tradition in the social sciences, particularly in anthropology. It is part of qualitative evaluation methods. It thus aims to overcome the limitations of quantitative surveys, which are based solely on statistical analyses: the latter provide an overall numerical picture of the results of a policy, but they say nothing about how it is implemented and the concrete difficulties that are responsible for its failures or unexpected effects. Direct observation, on the other hand, allows us to grasp the practical situations that constitute policy implementation on the ground: we then have a first-hand description of the implementation of a given programme, but also of the material conditions of its success or failure.

The direct observation of social practices has a long history. First of all, it is inseparable from anthropology and ethnology: when these disciplines fully constituted themselves as sciences during the 19th century, they progressively theorised ethnography as their main method of data collection. At the time, the aim was to study populations that were geographically and culturally distant. Observation made it possible to reduce the social distance with the subjects of the investigation through immersive research, which involved prolonged stays in the field, learning local languages, and a series of methodological precautions designed to avoid any ethnocentric judgement on the part of the ethnographer. At the end of the 19th century, and in a perspective closer to evaluation, the social surveys conducted in Europe among working-class or marginalised populations also used observation, which again was intended to reduce the social distance separating the ethnographer from the environment he or she was observing. Finally, in the 20th century, observation was used in sociology, and later in political science, to study ‘close’ objects (public services, political parties, organisations). The challenge is to ‘unfamiliarise’ these known practices, as the observer’s position invites us to decentralise our gaze and to question the causes and social mechanisms of activities that are taken for granted.

Within the qualitative family, observation is often combined with semi-structured interviews (see dedicated chapter on semi-structured interviews), both with administrative agents and with the publics they encounter. Here again, observation makes it possible to reconstruct what these interviews cannot say: first of all, it makes it possible to circumvent the self-censorship that informants often impose on themselves in interviews, particularly when it comes to talking about the quality of their work and the performance of their missions. It also makes it possible to describe precisely certain aspects of public policies that the evaluators and the evaluated would not think of mentioning in an interview. Local routines and habits, the practical organisation of work, postures and attitudes or non-verbal communication with users – and all that they reveal about the social relations and inequalities involved in the relationship between civil servants and their publics – are then made directly visible (Perret, 2008). This type of approach can be particularly useful when the policies evaluated target sensitive populations (precarious or socially marginalised people, people with disabilities, etc.), with whom interactions require specific skills on the part of civil servants: self-presentation, the ability to explain the administrative process or to manage the anxiety or anger of the public encountered.

Conducting ethnography requires special preparation (Becker, 2002). While it may seem easy to go to a place to observe it, it is necessary for the observer’s outlook to be informed, and thus to constitute the space(s) studied as a scene of observation. A great deal of theoretical and documentary work is therefore essential to identify the relevant observation sites: which offices to observe, in which location (rural, urban, rich or disadvantaged municipality)? What activities and dimensions should be focused on? Should one try to compare the same moment of policy implementation in different places, or on the contrary analyse the different stages of a single administrative chain? After answering these questions, the ethnographer must go to the field and confront the inevitable tension between closeness and distance from the respondents. Observation implies sharing the daily life of the people being surveyed over a long period of time, while minimising the distance that potentially separates one from them. It is therefore necessary to align one’s appearance, speech and body language as much as possible with that of the people being observed. Conversely, it is also advisable to regularly leave the field of observation in order to “retreat” into a space specific to the reflection on the activities observed: in this case, it is a matter of avoiding too strong an immersion in the practice, and thus of reinstating the external position of observation.

Throughout the observation, the observed activities are regularly recorded in a fieldwork diary, in written or recorded form. Although there is no standardised form or method for writing it, this diary must combine not only the description (of the places observed, with plans and sketches, and of the activities taking place there), but also the ethnographer’s reactions: surprise, indignation or sympathy in the face of the phenomena observed provide information on the sensitivity of the observer, but also on the divergent sensitivity of the people being observed: it highlights the production of local representations of what is ‘normal’, ‘acceptable’ or ‘problematic’, representations that are not (yet) shared by an outsider who discovers the situation. From a methodological point of view, recording one’s reactions during the observation also makes it possible to objectify them in order to analyse them, thus limiting the impact of the ethnographer’s subjectivity on their observations.

II. How is this method useful for policy evaluation?

As Stéphane Beaud and Florence Weber (2012) note, the adoption of the ethnographic method results from dissatisfaction with the discourse that a group – in this case an administration – holds about itself: it is a question of going beyond the official presentation of an activity, what the legal rules, instructions or presentation brochures say about it, to analyse the reality of its practice. Such direct observation can therefore take place ex post, at the stage of implementation of public policies, which we know often corresponds to a real re-elaboration of the policy by administrative agents. It is particularly justified when it comes to evaluating a policy format that is difficult to quantify (reception at an administration counter, for example, see next section). Such an approach makes it possible to observe the diversity of local investments in the same policy, and its adaptation to the local conditions of its implementation (specificity of the public, of the socio-economic or political context) or of the actors who carry it out (legacy of local routines specific to a department, an office or a municipality). Such a perspective opens up two potential evaluative logics: highlighting the local innovations of which street-level bureaucrats are capable in order to deal with situations not provided for by the texts, and also considering the multiple logics that can possibly cause a public policy to deviate from its stated objective. Typically, this involves evaluating how a policy and the material resources allocated to it adjust with the realities encountered on the ground, identifying the issues neglected during its design, and isolating the practices that need to be modified to enable public action to produce its full effects.

III. An example of the use of this method: the evaluation of the reception policy in public services

Although it is already old, the report submitted to the Prime Minister in 1993 on Les services publics et les populations défavorisées: évaluation de la politique d’accueil (Paris: la Documentation française, 1993) [Public services and disadvantaged populations: an evaluation of reception policies] is a good example of the usefulness of the ethnographic method for evaluation. It illustrates first of all the interest of observation in order to carry out a detailed approach to the question initially posed in 1990 by the Interministerial Evaluation Committee: in a context marked by the development of the theme of the modernisation of public services, the challenge was to evaluate the capacity of local public service counters to effectively deal with the difficulties encountered on a daily basis by the most precarious populations. Such an analysis could not be carried out through a purely quantitative evaluation, nor by a simple interview survey: the objective was indeed to take an interest in interactions – that of state services located on the ‘front line’ with the publics who most depend on the benefit they allow – and to try to evaluate their quality – in particular to judge the capacity of users to effectively assert their rights. The aim was to examine the implementation of reception services, the quality of information provided to the public, the impact on the effectiveness of their rights, the possibility of implementing satisfaction indicators and, ultimately, the appropriateness of adopting selective reception policies, some of which would be adapted to disadvantaged groups.

This report also highlights the fact that observation is often combined with other methods to shed light on ethnographic findings and to connect them to more general statements on the observed administration: in this case, the qualitative survey is combined with a quantitative component (questionnaires sent to users to select them according to their socio-demographic characteristics). Within the qualitative component, the observations made at the counter were supplemented by qualitative interviews with users, reception staff and ‘social intermediaries’ (associations or civil servants from the social services who facilitate access to public services).

The research required the joint work of the administration’s inspection services and consulting firms or academic research centres (3 private firms and a university centre), and a preliminary work of identifying the relevant observation scenes: each fieldwork was prepared by a mapping of all urban services, which made it possible to identify eight public services considered central to the problem of reception (police, hospital emergencies, town hall, etc.) The localities surveyed were selected because of their pre-existing classification as “disadvantaged areas”.

These methodological choices are not without bias and illustrate in passing one of the difficulties of ethnographic research and the joint importance of the initial question, and of the observation protocol designed to answer it. In this case, the report concludes that it is necessary to adapt reception policies to disadvantaged populations, in particular by creating platforms or “public service centres” that bring together in the same place, within marginalised neighbourhoods, the offices of different public services (post office, town hall, etc.). These conclusions have been criticised by academics who have conducted their own ethnographic studies of precarious counter users (see Siblot, 2005; also Dubois, 2003): by focusing solely on the dependence of users on public services, the evaluation remains blind, in their view, to the multiple ‘coping’ strategies that precarious populations are able to develop in order to assert their rights, and which an in-depth ethnographic survey reveals. Similarly, the evaluation is accused of making an abusive generalisation by asserting the dominated nature of the users, whereas they are unequally endowed with different sorts of capital, particularly educational capital, and some of them may be in a position to interact on an equal footing with the reception staff.

IV. What are the criteria for judging the quality of the mobilisation of this method?

Ethnographic observation will be all the more useful if the observers have been able to carry out ‘casework’: in other words, to constitute the always singular situations observed in the field into ‘cases’ that can corroborate or invalidate a theory. The challenge is then to ’empirically delimit what is a problematic relationship between ideas and evidence, between theory and data’ (Hamidi, 2012). Maintaining this relationship requires ethnographers to pay constant attention to the practices observed in the field: they regularly bring up unexpected logics or themes, which must lead to enriching or modifying the initial theoretical question. This is an important issue in policy evaluation, where the initial design of the evaluation mission may be modified to avoid neglecting certain realities in the field (a problem raised in particular in the case of the reception of underprivileged populations in public services, cf. previous section).

The complexity of the ethnographic exercise then lies in the ability of observers to articulate, in the same research, cases of different status (Hamidi, 2012, referring to the extended case theory of the Manchester School). We can thus associate ‘exemplary’ cases for which we can expect, given the context and the populations concerned, that the theoretical hypotheses will be fully validated (to keep the previous example: a post office counter in a working-class neighbourhood of a neglected urban area), and ‘borderline’ cases in which they will only be partially confirmed (another counter located in a less isolated neighbourhood, or located in an area with closer community solidarity or a narrower network of associations). The various factors that can influence policy implementation are unevenly present in these different cases: bringing them together therefore makes it possible to identify with precision those that have a full impact on public action and those that are more secondary.

V. What are the strengths and limitations of this method compared to others?

As we have seen, direct observation makes it possible to grasp ex post the material conditions of the implementation of a policy on the ground, away from official presentations. The identification of observation scenes that illustrate different configurations of implementation of the same policy can allow for a particularly detailed evaluation of the effects of a given policy.

As we have also seen, observation is most often intended to be combined with other methods and complementary approaches. A classic criticism of direct observation concerns the possibility of generalising its results (external validity): observations, carried out in a specific area and necessarily situated, would only concern the local context they describe and would not make it possible to move from the micro-sociological scale to the macro-scale, that of a more global evaluation of the public policy under study. This objection has been partly overcome in recent work, which has emphasised the need to supplement ethnography with other methods, in order to connect the practices observed locally with their institutional framework and its history. This link can be established differently depending on the approach: in Vincent Dubois (2003)’s research on family benefit offices (CAF), the interviews conducted with the staff make it possible to link the observation of interactions at the counter with the career paths of the civil servants, and beyond that with the institutional conditions of their recruitment (absence of a clear definition of the counter staff’s mission and job description, etc.). On the same theme, Jean-Marc Weller’s research (1999) focuses on the material organisation of reception in administrations and what it reveals (budget cuts, withdrawal of the welfare state and a new managerial conception that turns users into ‘clients’) in order to link the interactions observed in the field to the global reforms of public action, of which they are the reflection.

Another limitation of the ethnographic method is the investment in time and personnel that it requires. While observation is technically inexpensive – it requires neither recording equipment nor computer processing of the data collected – it does require the presence of an observer, or more often a group of observers working in a concerted manner on several scenes and for long observation sequences (several months), alternating periods of ‘withdrawal’ and then ‘return’ to the field. The aim is to capture changes in practices (particularly when evaluating the implementation of a recent reform, which field officials are discovering and then gradually appropriating), but also, as we have seen, to allow the evaluators to regularly withdraw from fieldwork in order to compare their conclusions during the course of the survey and to clarify or modify the general observations they intend to make about the policy being evaluated. Although this long investigation period may therefore seem time-consuming, it is clear that it does not only refer to “field” work and observation: it also corresponds to a period of (re)drafting the final evaluation report and the general conclusions it will propose.

Some bibliographical references to go further

Beaud, Stéphane. 2010. Guide de l’enquête de terrain: produire et analyser des données ethnographiques. Paris: La Découverte, Grands Repères Guides.

Dubois, Vincent. 2003. La vie au guichet. Relation administrative et traitement de la misère. Paris: Economica.

Hamidi, Camille. 2012. “De quoi un cas est-il le cas? Penser les cas limites.” Politix, n°100, vol. 4: 85-98.

Jeannot, Gilles. 2008. “Les fonctionnaires travaillent-ils de plus en plus? Un double inventaire des recherches sur l’activité des agents publics.” Revue française de science politique 58, n°1: 123-40.

Siblot, Yasmine. 2005. « “Adapter” les services publics aux habitants des “quartiers difficiles”. Diagnostics misérabilistes et réformes libérales », Actes de la recherche en sciences sociales, 159, n°4: 70-87.

Weller, Jean-Marc. 1999. L’État au guichet. Sociologie cognitive du travail et modernisation administrative des services publics. Paris: Desclée de Brouwer.