Comparing physicians’ and patients’ reporting on adverse reactions in randomized trials on acupuncture—a secondary data analysis

Complementary and alternative medicine (CAM) therapies are widely used to treat diseases [1‐6]. Acupuncture in particular has been shown to be useful for chronic pain conditions [7‐11]. Several studies were performed that evaluated its efficacy, effectiveness, and safety [12‐14]. The rate of patient consultations for CAM treatments has been increasing [15]. As a result, the number of reported adverse reactions might increase, especially those which are not always avoidable, such as haematomas, nausea, vomiting, and aggravation of symptoms [13, 16]. Previous observational studies showed that acupuncture can be considered a safe therapy [13, 17, 18], although some case reports might give another impression [12, 19]. However, serious life-threatening adverse reactions of acupuncture, such as pneumothorax, are very rare [12, 13] but have been published in some case reports [20]. Acupuncture treatment for chronic low back and knee pain had been included in routine reimbursements by statutory health insurances in Germany since 2007 [21‐24].

Patients who are interested in receiving needle acupuncture treatment should be informed about possible adverse reactions for ethical reasons [12] and patient safety should have a greater priority in acupuncture training [14]. However, the type and frequency of adverse reactions are difficult to compare between the various studies evaluating acupuncture safety [12].

A limitation in many trials is that only health care professionals, especially the physicians, have the responsibility to document (serious) adverse events or adverse reactions [25, 26]. The health care professionals have the professional competence regarding the evaluation of adverse events or adverse reactions, whereas patients have good individual knowledge about their own safety in healthcare [27]. Only a few studies document the adverse reactions by both physician and patients, e.g. [28] with the result that frequency and severity can differ between physicians and patients self-reports.

The aim was to compare patients’ safety reporting with physicians’ safety reporting regarding the safety of acupuncture using data from several large acupuncture trials. Furthermore, associations between patient characteristics and reporting of adverse reactions were evaluated.

The present secondary data analysis is based on the Acupuncture in Routine Care (ARC) studies that evaluated the effectiveness of the addition of needle acupuncture treatment [23] compared to usual care only. In those trials, patients and physicians had to complete questionnaires to document safety parameters.

We evaluated the reporting of adverse reactions in a secondary data analysis of a large semi-randomized controlled clinical trial on acupuncture for chronic pain patients. We compared patients’ and physicians’ reports regarding the frequency of adverse reactions and evaluated their agreement. Overall, the patients reported on average three times more adverse reactions than their physicians. The most commonly reported adverse reaction was bleeding/haematoma for both patients and physicians, similar to a study by Witt et al. [13]. Despite this, many types of adverse reactions were seldom reported, especially life-threatening adverse reactions such as pneumothorax [13, 20, 43]. No or only slight chance-corrected agreements existed. However, differences in actual frequency did not necessarily result in differences regarding frequency category commonly used in product descriptions [37]. Moreover, we observed that the chance of reporting an adverse reaction either by the patient or the physician was higher for patients who had agreed to be randomized at baseline, i.e., who were willing to participate in an RCT, were female, and had a higher education degree. Various reasons might explain the difference between patients’ and physicians’ reporting. In general, the physician is equipped with more medical knowledge than the patient due to long-term medical training and professional experience, which can impact the reporting of adverse reactions, especially when the causality is vague. Indeed, it is feasible that the patient is best positioned to report his or her own symptoms [44].

The communication about the treatment and about its potential adverse reactions, the motivation and time for reporting, the disease treated, and the general educational background might also impact reporting. Furthermore, the method of documenting adverse reactions might have an impact. In our study, the patient’s question used to specify adverse reactions offered tick boxes for bleeding/haematoma and local inflammation as examples, whereas the physician’s included a free text answer to this question. This and the fact that these reactions might not be considered side effects by acupuncturists from the perspective of traditional Chinese medicine have possibly contributed to the differences in these categories. Although we included a comparably large number of cases and treatments with more than 44,000 chronic pain patients, conclusions regarding the specific frequency of adverse reactions in acupuncture should be drawn very carefully when they are only based on our present study. For the evaluation of acupuncture safety, other even larger studies were specifically designed, performed and published [13, 17, 45].

This study has some limitations. Firstly, we used secondary data from December 2000 to July 2004 which were not primarily designed to evaluate the differences between physicians’ and patients’ reporting of adverse reactions. The primary aim of the ARC trials was the evaluation of efficacy, while evaluating safety was only one of many secondary outcomes. We did not adjust for multiple comparisons.

A further problem is that the patients and physicians do not rate completely independent because many adverse reactions are invisible to the study physician and have to be reported by the patient to the physician first, and the physician might explain the definition of adverse reactions to the patient. Hence, physician’s reports based on the patient’s reports could validate the patient’s report, assuming the physician’s assessment can serve as the gold standard. However, this might cause under-reporting, whereas over-reporting of too many unjustified adverse reactions could cause difficulties when explaining the safety characteristics of the intervention in a real life setting. A further limitation of our study is that the assessment of adverse reactions was based on retrospective self-reports, which can be influenced by recall bias [46]. The lack of differentiation between adverse events and adverse reactions caused by acupuncture could be an additional reason for the differential reporting. Definitions according to WHO, FDA or EMA differ by nuance [35, 47‐49]. In the literature, these terms are sometimes used synonymously (e.g., [50, 51]). The exact definitions of reputable institutions are listed in the supplemental material (see additional file 2).

In the physicians’ questionnaire in this study, we included a definition for an adverse reaction that referred to its noxious and unintended character to separate it from an adverse event. In contrast, the patient’s questionnaire did not include any explanation to improve clarity and usability of the questionnaire and because only adverse reactions and not adverse events had to be reported by patients. The difference in the questionnaires may to some extent explain differences in reporting of some adverse reactions, such as bleeding/haematoma that is sometimes intended by acupuncturists or pain, but not the differences for adverse reactions such as vertigo or fatigue. For future studies, we recommend a similar application of written definitions for both the physician and the patient questionnaire. However, tick boxes or free text should also be applied in a similar way. Not only who assesses but also how the assessment is performed can cause large differences in reported rates as shown in an RCT by Bent et al. [52]. This study compared three methods (1. an open-ended question, 2. an open-ended, defined question, and 3. a checklist of 53 common side effects) to assess adverse events experienced by study participants. The percentage of patients reporting any adverse events was much higher in the group using the checklist (77%) than in the first (14%) or second group (13%). This demonstrates the complexity of reporting and standards.

Strikingly, most of the studies on the safety characteristics of acupuncture are either based on therapists’/physicians’ or patients’ reporting but not of both [53‐56]. Fromme et al. investigated the clinician reporting of adverse reactions during chemotherapy [57]. In the study, 37 men with prostate cancer reported their adverse events, and the agreements with the study physicians using Cohen’s kappa was determined. The total Cohen’s kappa value was 0.15, which represents slight agreement, and was similar to our results. For rheumatoid arthritis, the reporting of adverse drug events between patients (n = 4246) and physicians differed; even for serious adverse events, the agreement was only 37% [58], whereas patients reported more events, which is similar to our results.

In a study comparing adverse events reported in post-discharge patient interviews with adverse events detected by medical record review, the agreement for adverse events (kappa = 0.20) and serious adverse events (kappa = 0.33) was low and comparable to our agreement results [59]. In contrast, in an oncology study in 2005, the agreement of 400 patients with their clinicians was higher (kappa up to 0.5) [60]. Especially, for observable reactions, the agreement was higher than for subjective ones [60].

A standardized reporting and documentation of both adverse events and reactions is essential [35, 61, 62]. For drug safety, the FDA developed a reporting system in 1998 [63]. For non-interventional pain studies, there are guidelines as CONSORT or ACTTION to document adverse reactions, but even these guidelines do not provide a high degree of detail [64]. In oncology, there are currently some documentation tools to combine the analytical and professional side of physicians and individual patient’s side using quality of life, symptoms, and patient-reported outcomes to enhance patient-clinician communication and to enable early detection of toxicities [60, 65]. In a British acupuncture study, the adverse events were monitored in a standardized way with self-reports by patient at each acupuncture session [66]. However, the documentation of safety by patient reports is still not standardized in clinical trials and health care. Even for the obligatory adverse drug reaction documentation, various information systems are used [67]. Standardized electronic web-based documentation software or intuitive mobile apps in contrast to classical methods (phone, questionnaire) could support the complete and harmonized documentation of adverse reactions [67, 68]. Further, it is important to differentiate between adverse events and adverse reactions and to evaluate a possible causal link to the intervention.

We think that both patients’ and physicians’ reports should be included when evaluating safety aspects of a medical intervention while electronic documentation tools might support this. Patients (or their relatives) can play an important role in signalling safety aspects in clinical trials as well as in routine care [62, 69] and can help the patient-centred approach in the future.

Regular communication between the physicians, other clinical staff and patients and the standardization of documents, including clarification of definitions, might help to minimize differences.

In our study, patients’ and physicians’ reports of adverse reactions of acupuncture differed substantially, possibly due to differences in patients’ and physicians’ questionnaires and definitions. The use of frequency categories has been shown to be useful and able to compensate for reporting differences. For the assessment of safety parameters, we strongly support the inclusion of both patients’ and physicians’ reports while ensuring standardization of data collection and definitions.