A new impetus for guideline development and implementation: construction and evaluation of a toolbox

Based on an analysis of barriers to the development and implementation of guidelines[8], a long list of potential topics for tools was composed. For each potential topic, we searched the literature for existing supportive materials. To prioritize the most relevant topics, we conducted a web-based survey among national guideline developing organizations and opinion leaders in the field of guideline development and implementation (Table 1). One hundred and forty-three online questionnaires were sent out and 74 questionnaires were returned, yielding a response rate of 52%. Responders were fairly equally divided between the approached professional groups. For each topic, the questionnaire included questions on awareness of existing guidance or tools; use of these materials in daily practice; and the extent to which there was a need for new supportive materials or tools. A nine-point response scale was used with answers ranging from ‘strong need’ to ‘no need.’ They were also asked to provide a top-three ranking of most relevant topics for the development of new tools. Finally, participants were asked to suggest additional topics that were not included in the overview of potential topics. In September 2010, we organized a national invitational conference to reach consensus on the topics for new tools. At this conference (attended by 49 responders to the online questionnaire), the results of the survey were presented. Problems as well as possible solutions on topics that scored a strong need for new tools were subsequently discussed in small groups. Finally, consensus was reached about a top ten of topics, added with two topics preselected by the funding organization of our project (cost-effectiveness in guidelines, shared decision making in guidelines). Table 2 provides an overview of the topics that were selected for the development of new tools. Table 2 shows that the selected topics cover the entire guideline development project: three topics specifically relate to the preparation phase of guideline development, three topics pertain to the development phase, and four topics refer to the completion phase. Two topics cover the entire guideline development process, i.e., organization and cooperation in multidisciplinary guideline development and project management in guideline development.

In October 2010, all Dutch organizations involved in guideline development, revision, implementation, and monitoring (n = 24) were asked to delegate professionals with specific expertise on one or more selected topics for tools. Thirteen working groups, each consisting of four or five people from different organizations (Table 1), were composed for the development of the tools. Each working group appointed a chairman, met four to six times face-to-face, and arranged some additional contacts by e-mail, telephone, or video-conferencing. After six months, each group delivered a draft version of the tool. This draft tool was sent to all organizations involved and a large group of experts for comment and review. One hundred and fifty experts submitted valuable suggestions for improvement. The received comments were analyzed and integrated in a next version of the tool.

National organizations involved in the development and implementation of clinical practice guidelines were approached to test the tools. Considering their current and future guideline projects, representatives of those organizations selected specific guideline projects for pilot testing of the tools. In 40 guideline projects selected, one or more tools were tested. In total, 18 organizations were involved in the pilot testing of the tools (Table 1). At the start of the evaluation, the project leader and chairman of the guideline group received detailed information on both the content of the tools and the aims of the pilot test. Halfway through the evaluation process, the project leader of the guideline project was contacted by telephone to inquire about the status of the evaluation of the tools. At the end of the evaluation process of the tools, a one-hour face-to-face meeting with the project leader, chairman and one or two other members of the guideline group was arranged. During this meeting, the following topics were discussed: experiences with the use of the tool and intentions to use the tools in future guideline projects; suggestions for improvement of the tool; and suggestions for dissemination and implementation of the tool. Based on the information yielded in these meetings, the tools were revised and their final versions were produced. With the exception of the tool on the prevention of improper influence due to conflicts of interest, the pilot testings of the tools were qualitative evaluations. In order to evaluate the conflicts of interests tool, individual members of guideline projects were approached with the request to complete a web-based questionnaire on their experiences and satisfaction with completing the ‘Declaration of Interests’ form. The participants answered the questions by marking their responses on a five-point Likert Scale.

The intention to use the tools in future guideline projects varied among the participating experts and depended on a number of factors, such as the availability and satisfaction with current procedures, the experience of key figures in the guideline development group and the time needed to apply the tool. Most experts involved in pilot testing of the tools had the intention to use one or more tools in future guideline projects, or at least specific parts of the tools that appeared to be most helpful for them.

After pilot testing of the tools, all tools underwent revisions on their content. Some tools were only slightly adapted to clarify their content, whereas a few tools were revised thoroughly based on the experiences during the pilot test. The scope of the tools on process management and project management was narrowed: the content of the tools was restricted to guideline projects, and general information on process and project management was removed. The tool on implementation of guidelines was aligned with the model for effective implementation of Grol et al.[9].

Experts involved in pilot testing of the tools proposed the following suggestions for dissemination and implementation of the tools.

The questionnaire was sent to 112 members of eight different guideline projects. From the approached members, 50 people (45%) filled in a questionnaire. Ninety percent of the respondents answered that the various relations and interests mentioned in the ‘Declaration of Interests’ form were completely clear for them. Nearly all respondents (94%) considered the form as an exhaustive and comprehensive document that would not miss any relations or interests. The respondents had differing points of view on the contribution of the ‘Declaration of Interests’ form to the actual prevention of improper influence. About one-half of the respondents (56%) answered that the form will enlarge both the awareness and the transparency on improper influence due to conflicts of interests, but considered this as insufficient for the effective prevention of it. Finally, ten respondents (9%) answered that the extent to which improper influence due to conflicts of interests occurs is not measurable with this form. Nevertheless, the large majority of respondents (85%) considered completion of the form as a first important step towards transparency and clarity on possible relations and interests and a useful document for assessment of the risk for improper influence due to conflicts of interest.

Twelve tools on guideline development, implementation, revision, and monitoring were produced, according to the following format: title of the tool; background; objective of the tool; content of the tool; and relevant literature. The tools differ in terms of structure: some tools include a step-by-step plan or a checklist, whereas other tools have a more descriptive character. Table 3 summarizes the main characteristics of the tools, i.e., the title of the tool, the objective(s) of the tool, and the structure of the tool. The full-text versions of the tools are available in Dutch and in English via a comprehensive website on guideline development and implementation (http://​www.​ha-ring.​nl).

Ethical approval was not needed for this study.