Content area experts as authors: helpful or harmful for systematic reviews and meta-analyses?

Who is a content area expert?

For this paper, we pragmatically define a content area expert as it is most often done—namely, a specialist clinician in the area under review or an author of an included study. Depending on the topic, different specialists may qualify as experts. For example, in a review of cancer screening, experts could be oncologists, radiotherapists, surgeons, radiologists, pathologists, or general practitioners. In can be hard to agree what constitutes a content area expert in some types of reviews, such as studies of general health checks, which involve testing for many different diseases and risk factors,6 and the use of biomarkers for diverse diseases.7

Potential benefits of including content area experts

Content area experts may have inside knowledge of unpublished trials or data or about things that went wrong, such as unsuccessful blinding or under-reporting of harms, that are not apparent in trial reports or subsequent letters to the editor. They are familiar with what is of current interest in their field and what questions are considered most pressing to answer. They could also mould the review in a way that makes it more interesting and informative.

However, it does not follow that content area experts need to be authors. Moreover, when expert agreement on important issues is low, it may be unhelpful to rely on one or two experts as coauthors, as they may have personal prejudices and idiosyncrasies.8 And if many experts or professional groups and societies are involved, the methodologists’ contributions may be drowned in a sea of expert opinions. If the special knowledge of experts is needed, it can be obtained by inviting them to stakeholder meetings2 3 4 or by asking them to peer review the protocol or the review.

For reviews involving meta-analyses of raw, individual patient data, it is common practice that investigators from the primary studies coauthor the review. Involvement of these experts is considered essential in understanding how the studies were conducted, whether there are any peculiarities in the data, and to correct inconsistencies and errors that may arise from logical queries in the datasets. Their author role is expected given that they “control” the studies and the data. However, if raw data become available to the public, bodies such as the Organisation for Economic Cooperation and Development, the European Commission, and the US National Institutes of Health agree that coauthorship cannot be a condition for using the data.9

Potential harms of including content area experts

The strong opinions specialist authors often have can make it difficult to perform unbiased systematic reviews, and the conclusions reached from a certain set of results can be particularly unpredictable. Many content area experts write opinionated non-systematic reviews or commentaries, and the stronger the expertise, the stronger the prior opinion, the lower the quality of the reviews, and the less time is spent on preparing them.8

When convincing randomised trials or systematic reviews find results that invalidate expert based practice, there is always a flurry of reviews, editorials, and letters from content area experts that try to refute, or even denigrate, the evidence. This wholesale editorial assault has been studied empirically—for example, for percutaneous coronary intervention in stable coronary disease,10 hormone treatment in postmenopausal women,11 and mammography screening.12 In the case of hormone therapy, access to industry documents showed many of the reports were ghostwritten by industry.13 14

Furthermore, depending on what type of expert is involved, the review may go down different paths and come to different conclusions. A survey of urologists and radiation oncologists, for example, showed that, for prostate cancer, the experts were more likely to recommend the treatment in their own specialty than the other treatment, independently of the Gleason scores and prostate specific antigen levels, even though their estimates of the risk of impotence and incontinence after radical prostatectomy and radiotherapy, respectively, were very similar.15

Even experts in the same discipline vary greatly in their opinions. A group of 57 medical specialists met to develop a guideline for a common and important intervention and were asked to write down their prior beliefs about the probability of a particularly important outcome. The answers ranged from 0-100% with clustering around 5% and 80%.8 16

Experts also have highly varying opinions about the overall scientific quality of review articles. In a study involving 36 reviews, the intraclass correlation coefficient was only 0.23 (95% confidence interval 0.03 to 0.45) for content area experts, whereas it was 0.79 (0.65 to 0.87) for experts in research methodology.8 Opinions about the rigour of primary research also vary widely, and experts tend to praise the papers that provide the results that confirm their beliefs.12

A further point is that research studies are often biased by their design, analysis, reporting, and interpretation,1 12 and the bias may have been introduced by content area experts who share the same views as those experts who might be considered as authors for the systematic review. It therefore doesn’t make much sense to include such experts when trying to unravel the hidden bias in study reports, unless one is reasonably certain that these experts also aim at getting to the truth rather than protecting the prevailing beliefs or income in their specialty. We would not want to be guided by people who have an interest in concealing uncomfortable evidence, and clinicians, for example, find it particularly difficult to acknowledge the harms their interventions may cause.12 17

When reviews include authors of the primary studies—for example, in meta-analyses of individual patient data—the situation can be particularly difficult, as we have both experienced.18 19 Primary authors are likely to defend their results and see the meta-analysis as an opportunity to advance their views. When presented with the results of a heterogeneous meta-analysis of the association between high levels of insulin growth factor and risk of prostate cancer that included their data, primary authors of studies with significant results concluded that the meta-analysis showed a strong effect, whereas methodologists concluded the effect was null or tiny (odds ratio <1.20).18

Box 2 shows some examples from our experience.12 19 20 We use them to show some of the problems that can be encountered.

Conclusions

The risk of introducing bias in a review may be increased when content area experts are included as authors. We therefore recommend that, by default, teams performing systematic reviews and meta-analyses should not include content area experts as authors. If this is considered indispensable, the inclusion of such people should be carefully justified. We agree with the US Institute of Medicine2 that experts with financial conflicts of interests should not be included as authors.

The Cochrane Collaboration states in its handbook that review teams must include expertise in the topic area being reviewed and include, or have access to, expertise in systematic review methodology.1 We suggest turning this recommendation around: review teams should include expertise in systematic review methodology and have access to expertise in the topic area. The importance of safeguarding the independence of researchers performing systematic reviews cannot be overstated.25 26 27