The evaluation of an evidence-based clinical answer format for pediatricians

The online survey was developed by an evaluation research consultant, based on initial input and ongoing review by the research team. The survey addressed several key aspects of the CA format: overall value and efficacy, adequacy of detail provided in sub-sections, suggestions for both format improvements and suggestions for content of future CAs (Additional file 2). An item was also included to determine the Net-Promoter Score (NPS), a tool used to measure customer satisfaction. NPS is derived from a single question that asks respondents to indicate - on an 11-point scale anchored at 0 (not at all likely) and 10 (very likely) - how likely they are to recommend a product to a friend or colleague [13‐15]. The NPS is calculated by subtracting the percentage of individuals who are unlikely to recommend a product, referred to as "Detractors" (0-6 on the 11-point scale), from the percentage of individuals who are highly likely to recommend it, called "Promoters" (9-10 on the 11-point scale). A bad recommendation to a colleague is associated with a potential risk to reputation, and as such, most individuals are only likely to recommend a product if they are extremely confident in its quality (Ibid). An NPS that is positive is beneficial in that it represents a potential increase in awareness and use of a product or service.

Quantitative data analysis consisted primarily of descriptive statistics. In addition, for items that asked respondents to comment on the level of detail provided in various sections of the CA, a binomial test was performed to determine whether the proportion of participants who thought there was too much detail (responses of "4" and "5" on the 5-point scale), or too little detail (responses of "1" and "2"), was significantly different from 50%.

A content analysis was performed on the qualitative survey data. For each qualitative question, coding categories were developed by examining participants' responses for recurring patterns. Individual comments were then coded into the appropriate response category by a researcher.

Characteristics of our 83 survey respondents are shown in Table 1. On average, respondents had been practicing pediatricians for 17.5 years (SD: 9.78), with the greatest proportion of respondents having practiced between 6-10 and 16-20 years (23% each). Almost all respondents worked in a clinical setting (96%) and over half were engaged in teaching (60%) or research (55%). Forty percent of respondents were involved in clinical care, teaching and research, and 28% were involved in clinical care only. The most common specializations were general pediatrics (28%), hematology/oncology (21%), and pulmonology or respiratory medicine (11%).

Overall, respondents found the CA useful (Table 2). Most respondents found the CA useful or very useful in presenting evidence (93%), agreed or strongly agreed that the CA contained the type of evidence necessary to support clinical decisions (88%) and that they would likely make use of CAs in the future (83%), and indicated that they would likely make use of CAs instead of full-length Cochrane reviews (58%). The positive survey responses were supported by respondents' written feedback: four times more respondents provided feedback that was positive (e.g. "this is excellent - the idea is good and the structure both simple and informative") as opposed to negative (e.g. "lazy way of presenting information").

A main purpose of our survey was to identify whether each section of the CA contained an appropriate level of detail (Table 3). Between 65% and 89% of respondents felt there was an appropriate level of detail in each section of the CA, with more than 75% of respondents thinking the amount of detail in the 'Question', 'Answer' and 'Included reviews' sections was appropriate. For respondents who did not think the level of detail in each section was appropriate, a significant number thought there was too much detail in the 'Question' (p = 0.04) and 'Linked figures and tables' (p = 0.0002) sections and too little detail in the 'Limitations' section (p = 0.008).

Respondents were asked qualitative questions to determine what type of information they felt should be added to or removed from each section of the CA. Most comments related to four recurring themes in the 'Results' section: 1) there should be additional information about the population (location of trials, ages of participants) and intervention (dose and schedule); 2) the word 'significant' should be replaced with the magnitude of the effect; 3) information about heterogeneity should be provided; and 4) statistical terms should be spelled out in full prior to being abbreviated. With the exception of respondents wanting less detail in the search strategy, the qualitative comments overwhelmingly suggested that our CAs should be slightly more detailed. One respondent stated: "I find myself just wanting more information before I view something like this and make a potential change in the way I approach a given clinical situation." Four respondents also shared the opinion that adding "a short concluding remark on what the authors recommend... would be useful."

Regarding questions related to the format of the CA (Table 4), 82% of respondents agreed or strongly agreed that the layout of the CA was an effective way to present content and that it allowed them to quickly locate critical information. Five of nine qualitative comments related to the CA format suggested minor improvements, such as providing links at the end of each section to return to the top of the page. Sixty-five percent of respondents were familiar with the use of GRADE assessments (Grading of Recommendations Assessment, Development and Evaluation), and only 61% of these respondents (40% overall) agreed or strongly agreed that including GRADE assessments in the CA would greatly enhance the quality of the evidence.

Lastly, we examined the likelihood of respondents recommending CAs to their colleagues and calculated an NPS of +32.6 from these data (Table 5).

To decide whether to alter the level of detail in any one section of the CA, we created a three-step decision rule. First, we determined the sections in which less than 75% of respondents felt there was an appropriate level of detail (Table 3), and then we carried out a binomial test as described in the Results section. Lastly, we used the qualitative data to determine which types of changes to implement. The binomial test indicated that the amount of detail in the 'Limitations' and 'Linked figures and tables' sections potentially needed to be changed. Upon review of the relevant qualitative comments, we did not find clear direction for increasing the amount of detail in the 'Limitations' section as there were no patterns in the responses; thus we will not be changing this section. For the 'Linked figures and tables' section, the forest plots will be removed because qualitative comments indicated that forest plots are a less familiar presentation format. Furthermore, while the p-value for the 'Question' section was significant, respondents strongly believed there was an adequate amount of detail (89%); therefore this section will not be changed.

Participants also offered a variety of small, easily implemented suggestions that will improve the overall usability of the CA. We will be adding information on geographic location, type of study, age of participants, dose and schedule where possible, spelling out statistical terms in full prior to being abbreviated, and replacing the word 'significant' with the magnitude of the effect. The formatting changes include: adding links to the systematic reviews and trials included in the CAs, and adding a link to each addendum to return to the CA front page.

During development of the CA the team were uncertain whether or not to add GRADE assessments [16] into the format. Because only 65% of respondents knew what GRADE was, and considering the cost in time and resources compared to the benefit of adding GRADE, we decided not to include GRADE in the CA format. A new section will be added: 'Implications for practice.' This section will provide a treatment recommendation and will take into account the overall quality of both the systematic reviews and their included trials for all outcomes of importance.

Attempting to answer all clinical queries as they arise is an important component in evidence-based medicine [5]. In a study conducted in two emergency departments in the US, emergency medicine physicians attempted to answer clinical queries as they arose 81% of the time [18], and were reported to be successful 87% of the time. CAs that address frequently arising clinical questions may increase the chance of success.

Clinicians are often concerned with their ability to keep up to date with the constant flow of new literature relevant to their practice [2, 5, 7]. Cochrane and other systematic reviews sift through all the available evidence in order to summarize the results of a specific clinical question but they still take substantial time to read, whereas a one-page CA summary could provide a much quicker way to keep up to date.

Because we did not practice random sampling, our survey has a potential for response bias. Those contacted were likely to be more literate in evidence-based medicine because of the sources through which they were contacted, and all respondents reported experience in teaching or research. Furthermore, the CA tested was on respiratory child health, and only a minority of our respondents (11%) specialized in that area. However, we are testing the format and not the content of CAs, and the broad range of child health specialists will help ensure that our CA format is suitable for a wide variety of topics.

With the potential for several thousand participants, we received 83 completed online surveys. However, despite the low response rate, and potential bias from the ratio of respondents from various specializations, we found the survey results to provide coherent, clear information in our product refinement. Furthermore, 21% of respondents were hematology-oncology specialists, and 28% were general pediatricians, which does not reflect the ratio in practice. A possible explanation for this is related to our decision to measure the NPS; a strong Promoter may have sent our survey invitation to a large number of hematology-oncology specialist colleagues.

Lastly, for CAs to be useful we require an increasingly internet and e-resource savvy clinical population [2, 3]; however, clinicians also need to be trained in searching and evaluating information for quality and reliability, preferably as early as medical school or the residency stage [19].