Chlamydia screening in England: a qualitative study of the narrative behind the policy

In 1996, the Chief Medical Officer (CMO) convened an Expert Advisory Group to formally consider the establishment of a chlamydia screening programme in England. The Group’s formation was in response to mounting interest in chlamydia screening in the UK [14, 15], and promising observational studies of screening in other countries [16, 17]. At this time, the other scientific evidence to support the effectiveness of chlamydia screening comprised one trial of screening from the USA [18]. Data from genito-urinary medicine (GUM) clinics across England showed increasing diagnoses in young people although there were no population-based estimates of chlamydia prevalence.

In 1998, the CMO published a report which proposed the introduction of chlamydia screening in England. It recommended an opportunistic screening model, targeted towards young women, delivered in general practice and community sexual and reproductive health services. It also called for a research programme including a randomised controlled trial of screening in England. The NHS Health Technology Assessment Programme subsequently commissioned cross-sectional feasibility and acceptability studies [19]. The Department of Health also funded a pilot of opportunistic screening in two areas in 1999. These pilots – where GPs were paid to offer tests - reported high uptake in primary care (>50 %) and found over 10 % of young people tested positive for chlamydia [20, 21].

In 2001, the sexual health strategy for England 2001–2011 was published. This strategy highlighted serious problems in sexual health care, including four-week waits for urgent appointments in GUM and patchy provision outside specialist GUM services, with few GPs providing sexual health care (health promotion, advice or STI testing and treatment) other than contraception. It presented a vision of sexual health as an integrated programme, encompassing both contraception and STIs (including HIV) and proposed a new model for delivering sexual health services, where provision was expanded to primary care [22]. As part of this model, it proposed a national screening programme based on the CMO’s 1998 delivery recommendations, to be implemented in phases, with full rollout across England by 2008. Testing in GUM clinics was not part of this programme. In 2004, the Public Health White Paper accelerated the schedule, promising national provision by 2007 and announced funding to support implementation [23]. In the absence of empirical studies, modelling data estimated the level of coverage required to reduce prevalence [24]. National targets for screening coverage were first announced in 2005. From 2007, local areas were monitored against these targets.

The NCSP delivery model changed as the Programme expanded. By 2003, the target population included men. By 2004, testing was conducted outside of general practice and sexual health services in non-traditional settings such as pubs, clubs, sporting events and festivals. Implementation did not occur as planned: national provision was later than expected, numbers of people tested remained low, particularly in primary care, which accounted for less than 20 % of chlamydia test delivery, and amongst men, who account for less than 40 % of NCSP tests [25]. As a result, coverage targets were missed by a significant margin in 2007/2008. After 2008, chlamydia tests performed outside the NCSP (but not in GUM) were also counted towards coverage and targets were almost met [25].

From 2006, the validity of earlier trials was more thoroughly and widely questioned in systematic reviews, prompting more empirical research outside England to underpin screening policy [26, 27]. In addition, researchers have since given greater attention to the secular changes (eg lower rates of partner change as a result of HIV prevention messages in the 1990s) that may have explained the initial fall in chlamydia diagnoses following the introduction of screening programmes [28, 29]. Evidence remained equivocal: for example, in 2010 a trial in England of chlamydia screening reported a non-significant benefit of screening for reducing pelvic inflammatory disease [30]. This trial’s limitations also illustrated the significant challenges of delivering chlamydia screening even in a research context eg achieving uptake of sufficient numbers of young people [31].

We did not identify a consistent theoretical basis for the NCSP. However, two underlying themes emerged that underpinned the establishment and implementation of the NCSP. We refer to the first theme as ‘explicit’ because it is central to the stated aims of the Programme and the second theme as ‘implicit’ because it was not stated in policy documents but emerged from interviews.

Theme 1: The first theme corresponds to the explicitly stated aims of the NCSP, which are to reduce chlamydia prevalence and sequelae, in response to concerns about rising diagnoses (Figure 1, dark blue boxes). Diagnosis was made possible by the advent of new technology, which was highly sensitive to detecting chlamydia and in contrast to older tests, was much less invasive, often requiring a urine test only. As the illustrative quotes below indicate, he importance of advances in diagnostic technology for the introduction of screening was corroborated by evidence from interviews, and other contemporary documents.

The CMO’s report identified chlamydia as a public health problem and screening as the policy response. This position was widely supported by the scientific and clinical community in England:

Johnson, BMJ, 1996 [15]

Boag and Kelly, BMJ, 1999 [32]

" We’d been seeing chlamydia figures going up and up and there was a growing awareness that it was a major cause of pelvic inflammatory disease…. ectopic pregnancy and I suppose to my mind the trigger for all of this probably was the advent of molecular diagnostics, the idea that actually you could undertake testing using so-called non-invasive specimens. "

Interview, CMO Expert Advisory Group member (1996–2001)

While the CMO’s report acknowledged that data on prevalence of chlamydia sequelae were “incomplete” and “uncertain”, its bold statements that sequelae were “severe” and that management would result in “considerable health benefit” [2] went largely unchallenged in subsequent commentaries and letters, despite the lack of trial evidence from England at that time and the incomplete knowledge concerning the prevalence and natural history of chlamydia. Instead, discussions focused on the extent to which screening would be acceptable to young people and how it should be delivered. This is illustrated in the quotes below:

Theme 2: A second theme also emerged from the interviews (Figure 1, light blue boxes). It was not referenced in policy documents, and we therefore refer to it as an implicit theme. This was concerned with the contribution of chlamydia screening to advancing wider sexual health service delivery. As described in the timeline and in interviews, sexual health services in the late 1990s/early 2000s were in urgent need of serious investment:

Boag and Kelly, BMJ, 1999 [32]

"The Chief Medical Officer's plan for immediate action on Chlamydia trachomatis…is a step in the right direction, but it does not go far enough."

Letter from Opaneye, BMJ, 1999 (in response to Boag & Kelly editorial) [32]

Interview, Independent academic expert (1996–2011)

Letter from Macleod et al, BMJ, 1999 (in response to Boag and Kelly editorial) [32]

"[The NHS pilots aimed to find out] how feasible was it to ask people to pee in a pot [i.e. do a urine test]… for an STI they hadn’t actually gone along to ask about in the first place."

Interview, NCSP Steering Group member (1999–2008)

Interview, NCSP Steering Group member (2000–2004)

Rising chlamydia diagnoses were quoted by interviewees and widely in policy documents to support claims of increasing burden of sexual ill-health:

Third report of session 2002–03 on sexual health, House of Commons Health Committee, 2003 [33]

As shown in the timeline and corroborated in the interview data, in contrast to the previous decade, significant efforts were now successful in gaining political recognition of this problem:

Interview, Independent Sexual Health Advisory Group member (2002–2011)

Chlamydia screening was recognised as a vehicle to engage young people in discussions about their sexual health and an opportunity to drive increased access to services for management of STIs (white dotted-line box, Figure 2). This was evident from interviews and from an article by those leading the first English screening pilots:

Pimenta et al, BMJ, 2000 [34]

Interview, CMO Advisory Expert Advisory Group member (1996–1998)

Specifically, implementing a programme of chlamydia screening was expected to expand sexual healthcare in primary care and contraceptive services:

Interview, NCSP Steering Group member (2000–2004)

Interview, National policy maker (2000–2005)

Although the Government frequently refers to chlamydia screening as part of its service reforms, neither expanding access to sexual health care nor engaging young people in sexual health were stated aims of the NCSP. Similarly, this theme does not feature in the CMO’s report [2], despite interview evidence that it was discussed within the Expert Advisory Group:

Interview, CMO Advisory Expert Advisory Group member (1996–1998)

Interview, National Screening Committee member (1996–2007)

The aspiration to use screening in order to expand community sexual health services and to promote discussion of sexual health with young people therefore appears as an important, but largely implicit, influencing factor in establishing the NCSP.

Our analysis also suggests however, that this implicit theme influenced key implementation decisions. These decisions included men’s eligibility for screening, where there is little empirical evidence to guide decisions; most published randomised controlled trials (RCTs) have been conducted on women [18, 30]. The CMO’s Expert Advisory Group report (1998) initially recommended that screening should focus on women only. This proposal was adopted in the first phases of implementation. The CMO’s recommendation was based partly on considerations of feasibility, recognising that women are “are more likely to attend health care settings” [35]. However, the NCSP policy changed since the initial phases to recommend that programmes should screen men and women equally [36]. This was partly because studies conducted since the CMO’s report found chlamydia prevalence was similar in men and women [37, 38]. However, there was still no evidence that including men in the target population for the NCSP would be cost effective in preventing chlamydia-related harms. Our interviews and documentary evidence suggest that the policy change was based less on the potential to control chlamydia, and more to promote equitable engagement of men in sexual health:

Duncan and Hart, BMJ, 1999 [39]

Interview, NCSP Steering Group member (2000–2004)

Interview, NCSP Steering Group member (2002–2005)

Men’s Health Forum, 2005 [40]

The two themes evolved during implementation of the NCSP.

Theme 1: Following rollout across the country, the gaps in the evidence base to justify screening were now more widely recognised. Flaws in early RCTs and questions about the effectiveness of screening to control chlamydia and prevent reproductive ill-health led to questions about the “alacrity” with which “influential groups have adopted chlamydia screening” [41]. As the quote below shows, he gaps in the evidence surrounding the natural history of chlamydia became a central question for researchers.

Interview, Independent academic expert (1999–2011)

These questions also led to questions about the policy of funding chlamydia screening in England:

Theme 2: The focus of the NCSP’s monitoring was entirely on delivery of testing and managing infections so any wider effects were not formally captured. Evidence from interviews suggests that early in implementation, the NCSP drove integration of services to some extent and did contribute to expanding sexual healthcare delivery beyond specialist services:

National Audit Office, 2009 [3]

Interview, National policy maker (2001–2010)

There was some evidence that providers in new services used chlamydia screening as an opportunity to discuss sexual health with young people outside services.

Interview, Local implementer (2008–2011)

As the Programme expanded, pressure to achieve high coverage led to new services focusing solely on chlamydia testing. These services became divorced from mainstream care and offered little opportunity for sexual health promotion [3]. The National Audit Office reported in 2009 that 40 % of young people tested within the NCSP by did not receive sexual health advice.

Interview, National policy maker (2001–2010)

Interview, National policy maker (2002–2005)

Interview, Local implementer (2008–2011)

In addition, our interviews reflect the conflict between achieving testing volumes and providing integrated sexual health care through chlamydia screening. The following two quotes come from two people involved during a similar period of the NCSP’s development, both working to implement the NCSP at a national level. These show that some of those involved in implementing the NCSP at a national level stated it was unacceptable for health professionals to avoid discussing sexual health with young people. However, others minimised the input required from health professionals:

Interview, National policy maker (2002–2005)

Interview, National implementer Implementer (2001–2004)