A systematic review of observational methods used to quantify personal protective behaviours among members of the public during the COVID-19 pandemic, and the concordance between observational and self-report measures in infectious disease health protection

Throughout the COVID-19 pandemic, members of the public have been urged to engage in a set of behaviours intended to reduce transmission of the SARS-CoV-2 virus. These have included recommendations to practice frequent hand hygiene, avoid close contact with other people (‘social distancing’) and to wear a face covering to prevent spread through respiratory droplets [1, 2]. Although these interventions have been shown to be effective in reducing the transmission of SARS-CoV-2 (https://​www.​bmj.​com/​content/​375/​bmj-2021-068302), none of the interventions will work if people do not adhere to them or understand the messaging on when they should adhere [3].

To date, public engagement with recommended behaviours has primarily been monitored by behavioural scientists, public health agencies and national governments though the use of self-report questionnaires. Using self-report methods to collect data has many benefits. Self-reported data can be quick, easy and relatively inexpensive to obtain from large numbers of participants. The association between self-reported behaviour and other self-reported variables is also relatively straightforward to examine. For many outcomes, self-report can be a good proxy measure for actual behaviour [4]. For example, self-report can be a useful way to assess whether someone has been vaccinated or not [5]. For other behaviours, self-report may be less valid [6]. This may be particularly true for frequently performed behaviours that are difficult to remember (e.g. frequency of handwashing in the past 24 hours) or that would be socially undesirable to admit (e.g. breaking legally enforceable rules around self-isolation) [7, 8, 9].

In the context of COVID-19, regularly collected measures of behaviour that do not rely on self-report are rare. Notable exceptions include: mobility data based on mobile phone locations [10]; footfall data in city centres [11]; and official statistics on vaccine uptake based on electronic records [12]. Most of these examples relate to where people are located or whether they engage with health services.

There are fewer regularly collected data based on direct observation quantifying whether and how people engage with COVID-19 protective behaviours. To assist public health agencies in considering whether to collect more observational data, we conducted a systematic review of the use of observational measures of COVID-19 relevant behaviours. We focussed on studies that directly observed the performance of protective behaviours, excluding measures of mobility or location. Our aims were to assess: 1) the quantity of observational studies conducted during COVID-19; 2) the quality of these studies; and 3) the association between self-report and observational data. While we only included COVID-19 related studies for aims one and two, given a lack of data found during screening, we expanded our inclusion criteria for aim three and included studies relating to any infectious disease outbreak.

Twenty-seven thousand two hundred seventy-nine published papers were identified that included terms relating to COVID-19, and one or more terms relating to hand hygiene, face covering and social distancing. When the term ‘observational’ and related terms were added, 2589 papers were identified and these were screened for aims one and two, from which 105 were selected as potentially relevant to the review. Of these, 57 were excluded. A total of 48 studies met the inclusion criteria (Fig. S1). For aim three we screened 3331 papers, from which 133 were deemed potentially relevant following abstract screening. Of these, 21 were included in the review (Fig. S2).

Improving uptake of health protective behaviours is an important public health challenge, not only for COVID-19, but for infectious disease prevention more widely. Face coverings, hand hygiene and maintaining physical distance have all been identified as effective infection control strategies that have relatively few downsides in comparison to more far-reaching interventions such as society-wide ‘lockdowns’ [11]. Identifying ways to achieve good adherence to such measures requires that we are first able to measure adherence accurately. Though self-report can be a useful proxy for behaviour, our review suggests that academic research has become overly reliant on it, so much so that although we identified 27, 279 papers which included terms related to COVID-19 and to hand hygiene, face covering or social distancing, just 48 of these papers (< 0.2%) actually studied the behaviour in question objectively.

It is likely that several factors influence the way in which behaviour is measured. Speed, cost, ease and the ability to explore associations with other variables which may be best measured via self-report (such as, for example, trust in government, exposure to conspiracy theories or the perceived efficacy of an intervention) are all valid reasons for opting to use self-report over observation. Indeed, the importance of ease as a factor likely explains why only seven studies have attempted to assess physical distancing using observational techniques while 35 have looked at face coverings: it is difficult for an observer to judge distance between two people and to identify whether people need to maintain distance from each other (e.g. if they do or do not live in the same household), but easier to assess whether they are wearing a face covering. Nonetheless, our review also points to the dangers of over-reliance on self-report. The 21 studies that we identified which compared self-report and observed behaviour repeatedly demonstrated that self-report over-estimates hand hygiene behaviour, sometimes dramatically so, while evidence for the validity of self-reported distancing and face covering use is limited in the current literature. Although outside the date range for our search, two recent pre-prints support this point for hand hygiene and extend the evidence of a self-report gap to include face covering and distancing. One study demonstrated that self-reports of “always” wearing a face covering when in specific public spaces in a national UK Government-funded survey matched observed behaviour in those locations, but that an additional 23% of people reported “sometimes” wearing face coverings in these situations, something which could not be accounted for in the observations [80]. The other study, of a single university campus in the UK, found that while 68% of survey respondents reported always cleaning their hands while entering a university building, observation of the only entrance to the main campus building found the true rate to be 16% [81]. Reported and observed rates for distancing were also discordant (49% vs 7%) while the gap for wearing a face covering was smaller but still noticeable (90% vs 82%). While multiple factors may account for these discrepancies [6], recall bias and social desirability would seem most likely to lead to inflated estimates of behaviours such as hand hygiene and physical distancing. Although there are exceptions, with some individuals exercising continuous daily mask use, the relative novelty of face coverings for many people, the limited number of occasions they need to be worn during the day for many members of the general public and their greater salience and hence ease of recall may partly mitigate these biases and explain why self-reported wearing of a covering may be a more reliable measure of behaviour than self-reported hand hygiene or distancing.

Notably, the quality of studies that included an observational measure was generally good in most respects. The one exception was that relatively few studies provided a sample size justification. We suspect this is linked to the difficulty of setting a pre-determined sample size in advance of a naturalistic study. For example, it can be difficult to predict how many people will pass by an observer over a set period of time. The relatively high quality may reflect the tendency for authors who choose to take the difficult route of evaluating behaviour via an objective measure to have also considered other ways of maximising the quality of their study.

The COVID-19 pandemic witnessed an explosion in research covering every aspect of the crisis. Within the field of behavioural science, there has been a heavy focus on ways to promote behaviours believed to reduce infection transmission. Almost all of these studies have measured whether people say they have engaged in specific behaviours. Few have measured the behaviour itself. This is problematic. For hand hygiene, observed adherence tends to be substantially lower than estimates obtained via self-report. There are few studies that have tested the validity of self-reported face covering use or physical distancing, but these also suggest that self-reports tend to be biased. Future research in this field should make greater use of observational methods where possible and should carefully consider the validity of any self-report measure where this is not possible.