Review and evaluation of the methodological quality of the existing guidelines and recommendations for inherited neurometabolic disorders

For each of the 682 identified iNMDs, grouped in ten categories, the following electronic databases related to GLs and RCs were systematically searched on February-March 2015: PubMed (http://​www.​ncbi.​nlm.​nih.​gov/​pubmed); the National Guideline Clearinghouse (NGC, http://​www.​guideline.​gov); the Guidelines International Network (G-I-N, http://​www.​g-i-n.​net); the Scottish Intercollegiate Guideline Network (SIGN, http://​www.​sign.​ac.​uk); the National Institute for Health and Care Excellence (NICE, http://​www.​nice.​org.​uk). Additional publications were included after manually checking the reference lists of the identified relevant documents. The strategy used to identify the GLs and RCs is shown in Fig. 1.

For PubMed database, the following searching strategy was used for each of the 682 diseases:

#1: “Disease” [Mesh]

#2: “Disease”

#3: #1 OR #2

#4: Recomm* OR Management OR update [TI/AB] OR Therapeutic* [TI/AB] OR treatment* OR guideline* OR consensus OR standard OR criterion [TI/AB])) OR ("Disease Management" [Mesh] OR "Therapeutics" [Mesh] OR "Health Planning Guidelines" [Mesh] OR "Guidelines as Topic" [Mesh] OR "Practice guidelines as Topic" [Mesh] OR "Review" [Publication Type] OR "Guideline" [Publication Type] OR "Practice Guideline" [Publication Type].

#5: #3 AND #4.

Search terms: disease, recommendation, management, update, therapeutic, treatment, guideline, consensus, standard, criterion, disease management, health planning guidelines, practice guideline, review.

All the GLs and RCs published from 2000 to 2015 in the mentioned databases were selected by one of the authors (LC). Only English-written publications were considered. Duplicates of documents, which were found in more than one database, were manually removed from the selection. The initial search yielded unique records based on title and abstract (Fig. 1).

The final documents were then selected by two authors (LC and AGC) after reading the full text. First and updated versions of the same GLs and RCs were considered as distinct documents. Publications were excluded when the method used to formulate the GL or RC did not include a formal consensus process (such as the one reached through the Delphi Method or repetitive encounters) among independent professional groups or individuals. Moreover, GLs and RCs were not considered when they are applicable to one unique country. Reviews, algorithms and letters were also excluded. The selected unique records that responded to all the criteria for evaluation cover disorders related to:

Defects of synthesis and remodeling of complex lipids were not considered in the search, because many of them are emerging disorders described in the last recent years.

For each study the following data were extracted: year of publication, name of the covered disease, number of authors, number of countries involved, country of origin of the authors, number of affiliations, and topics (screening, diagnosis, management and follow-up).

The AGREE II instrument was used to assess the transparency, the methodological quality and the rigour of the selected GLs and RCs. It consists of 23 key items that address six quality domains: (1) scope and purpose, (2) stakeholder involvement, (3) rigor of development, (4) clarity of presentation, (5) applicability and (6) editorial independence. Two additional items concern the overall judgment of the GL (Overall Guideline Assessment) (Table 1).

The detailed criteria for each item are available in the user manual for AGREE II tool (http://​www.​agreetrust.​org/​). Briefly, for each document, the twenty-three items were rated on a 7-point scale (1–strongly disagree to 7–strongly agree) by two trained independent reviewers, experts in the field of IEMs and belonging to clinical, biochemical or genetic professional categories. One reviewer (LC) trained all the reviewers and rated all the documents, in order to provide a minimum variability between appraisals. A score of 1 was given when little or no relevant information was presented and a score of 7 was given when the statement met all criteria. Disagreement between reviewers (defined as ≥3 points difference in the score assigned by the appraisers to the same item) was resolved through consensus. To achieve consensus, the two reviewers shared the rationale for their appraisal and helped by the instructions provided by AGREE II instrument, they reached an agreement. According to AGREE II instructions, domain scores were calculated as (obtained score –minimum possible score)/(maximum possible score – minimum possible score) where "obtained score" is determined by summing up all the scores given by the appraisers for the individual items in a domain. All the final domain scores were entered into an Excel spreadsheet. Since the AGREE II manual does not provide guidance regarding how to interpret scores, to determine the grade of recommendation of the GL/RC we used a method previously applied by other authors with some minor modifications [10].

A GL or RC is “strongly recommended (SR)” when all the six item scores were ≥50 % or five item scores were ≥50 % and one item score was between 40 and 50 %. A guideline is “recommended (R)” if the overall quality assessment (OQA) score was ≥50 %. A guideline is “not recommended (NR)” if the OQA score was < 50 %.

The term GL and RC was assigned to the documents on the basis of the definition that the developers used to define the guidance they elaborated. However, all the articles underwent the same AGREE II appraisal process, regardless if they were GLs or RCs.

Data analysis was performed using the SPSS 22.0 software. The Kolmogorov-Smirnov test was applied to assess the data distribution. The ANOVA test was applied to compare score means of domains and individual questions between the established guideline groups. To study differences between GL groups, either parametric tests (if they follow a Gaussian distribution, Bonferroni Test) or non-parametric test (Games-Hewell) were applied. Pearson correlation test was applied to search for correlation between year of publication and number of GLs or RCs, and overall quality. Finally, the correlation between the quality of a GL and the number of authors, countries and affiliations involved was assessed. Statistical significance was defined as p < 0.05.

Table 2 summarizes the information relative to the fifty-five identified GLs and RCs encompassing iNMDs.

GLs and RCs were identified for almost all the subtypes of iNMDs, except for disorders of purine, pyrimidine and other neurotransmitter metabolism (group B), lipid metabolism (group D) and peroxisomal disorders (group J). The groups of disorders highly differed in the number of associated GLs and RCs, since documents encompassing disorders of vitamin and non protein cofactor metabolism and transport (group E), porphirin and hem metabolism (group F) and energy metabolism (group H) were covered by a small number of GLs and RCs (n = 2, n = 1, n = 3, respectively). In contrast, the disorders associated with lysosomal and lysosomal-related organelles were the most encompassed by GLs and RCs (n = 27, Fig. 2a). These numbers did not necessarily correlate with the number of disorders in each group. For instance, the number of disorders associated to lysosome and lysosomal-related organelles (group I, n = 115) was lower than the number of disorders of energy metabolism (group H, n = 155), even though the number of GLs and RCs was ninefold higher in the former group (n = 27 and n = 3, respectively, Fig. 2b).

Six of the identified documents were about groups of disorders (11 %), while the remaining 49 pubblications focused on one unique disorder and globally covered twenty different diseases (Table 2). Moreover, 52 % of the considered pathological conditions were covered by only one document. In contrast, GLs and RCs about Gaucher disease, Fabry disease, and mucopolysaccharidoses (MPSs) were reported in seven (Gaucher disease) and eight (Fabry disease and MPSs) different publications (Additional file 1: Table S1 and Fig. 2c).

Although the number of new documents/year has been quite stable all over the last first decade, the overall frequency of GLs and RCs about iNMDs has significantly increased over time (Fig. 3a and b).

The analysis of the country of origin of the authors revealed that 29.1 % of the GLs and RCs were from Europe, 23.6 % from USA and the remaining 42.3 % documents were the result of a collaboration among authors coming from different continents or American countries (miscellanueous origin) (Fig. 3c).

Finally, 87.3 % of the GLs and RCs covered more than one topic and dealt principally with the diagnosis (80 %) and management (91 %) of the disease. Screening and follow-up-related issues were encompassed by 34 % and 55 % of the documents, respectively. No GLs and RCs have been developed for the follow-up of disorders of porphirin and hem metabolism (group F), mineral absorption and transport (group G) and energy metabolism (group H, Table 2 and Fig. 3d).

The level of recommendation of the GLs and RCs was determined by the number of items scoring ≥50 %. The AGREE II analysis revealed that 25 % of the identified GLs were strongly recommended (SR), 64 % recommended (R), and 11 % not recommended (NR, Fig. 4a). All the documents had at least one item scoring ≥50 % (Table 3). We excluded from the subsequent statistical analysis the unique document relative to disorders of porphirin and hem metabolism (group F).

In this domain, the appraiser is invited to judge the overall quality of the GL and to indicate whether she/he would recommend it for use.

The range and mean ± SD of the overall quality assessment (OQA) score for this domain were 25–92 % and 63 % ± 15 % (Table 3 and Fig. 4b). Interestingly, although we did not find any statistically significant differences among groups, the OQA of the GLs and RCs about iNMDs increased over the years (Fig. 4c).

A further 3-point scale (1 = not recommended, NR; 2 = recommended with modifications, R + M, and 3 = recommended, R) was introduced in the analysis, providing an additional overall judgment on whether the GLs or RCs should be recommended for use (Table 3). For this item, the range and mean ± SD of the overall score were 1.5–3 % and 2.2 % ± 0.4 %. Almost all the GLs and RCs were recommended, since they scored ≥2, except for six documents (11 %) that were not recommended. On the other hand, appraisers considered that 83 % of the GLs and RCs required modifications. Of note, two out of three documents recommended without modifications by both appraisers belonged to the group of amino acid and organic acid metabolism (group A).

The quality of a GL is supposed to improve when it is developed by experienced experts coming from different countries, or distinct professional categories. For this reason, we next wondered whether the overall quality of the GLs and RCs could depend on the number of authors, countries or affiliations to which authors belong. Figure 5a-c shows that a direct correlation between the OQA and these parameters could not be determined, although a non statistically significant increase in the OQA was observed for higher number of authors and affiliations.

This domain considers whether the overall objectives of the GL, the health questions covered by the GL and the population whom the GL is directed to are specifically described. The range and mean ± SD of the overall score for this domain were 56–100 % and 78 % ± 18 %. Table 3 and Fig. 6a report the score recorded for each GL and the overall score obtained by the different groups of disorders. The scores were comparable for all the considered groups. None of the GLs or RCs scored <50 %.

This domain evaluates whether the GL development group includes individuals from all relevant professional groups, the views and preferences of the target population have been sought, and the target users of the GL are clearly defined. The range and mean ± SD of the overall score for this domain were 8–92 % and 48 % ± 23 % (Table 3 and Fig. 6b). Almost half of the overall GLs and RCs, as well as all the documents related to disorders of vitamin and non protein cofactor metabolism and transport (group E) and mineral absorption and transport (group G), scored < 50 %. In addition, we also found statistically significant differences in the scores obtained by the different groups in item 4, which refers to the composition of the GL development group (Fig. 7a). For item 5, referring to the involvement of the target population in the elaboration of the GLs, 75 % of the documents had a score between 1 and 2, indicating that most of them did not consider patients or public in the process (data not shown).

This domain relates to the method used to search and select the evidence and to formulate the GL. It also focuses on the health benefits, side effects, and risks that should be considered when formulating the recommendations. The range and mean ± SD of the overall score for this domain were 13–95 % and 48 % ± 22 %. (Table 3 and Fig. 6c).

Whereas all the documents for disorders of amino acid and organic acid metabolism (group A) obtained a mean score ≥50 %, thirty-one GLs and RCs (56 %) presented scores < 50 % for this domain.

The statistical analysis of the distinct items revealed that the seven groups of disorders obtained different scores in items 7, 8, 9, 10 and 13 (Fig. 7b-f). Item 13 considers whether the GL has been externally reviewed by experts prior to its publication. For this items, only 30 % of the documents obtained a score > 3, indicating that the majority did not undergo an external revision prior to submission for publication or did not provide sufficient information. Item 14 judges if a procedure for GL updating has been established and 73 % of the documents provided scarce information (score ≤3, data not shown).

This domain examines whether the RCs are specific and unambiguous, the different options for management are clearly presented and key RCs are easily identifiable. The range and mean ± SD of the overall score for this domain were 31–100 % and 74 % ± 20 %. Table 3 and Fig. 6d show that all the groups of disorders presented a comparable score for this domain and no statistically significant difference was observed. Only two documents (3.6 %), belonging to the disorders of energy metabolism (group H) and lysosomal and lysosomal-related organelles (group I), presented scores < 50 %.

This domain examines whether the GL describes facilitators and barriers to its application, explains how the RCs could be put into practice and considers the potential resource implications of applying the RCs. The range and mean ± SD of the overall score for this domain were 8–73 % and 39 % ± 16 % (Table 3 and Fig. 6e). Collectively, 76 % of all the identified documents had a score < 50 % and all the GLs and RCs about mineral absorption and transport (group G) and energy metabolism (group H) obtained scores < 50 %. Analyzing the distinct items, we observed that 38, 58, 80 and 24 % of the documents achieved scores ≤3 for item 18, 19, 20, and 21, respectively. In particular, for item 20 (potential cost impact of the GL) the majority of the identified GLs did not sufficiently consider the cost effectiveness or implications for budget of applying the recommendations (data not shown).

This domain assesses whether the views of the funding body have not influenced the content of the GL and whether the competing interests of GL development group members have been recorded and addressed. The range and mean ± SD of the overall score for this domain were 0–100 % and 41 % ± 35 %. (Table 3 and Fig. 6f). No statistically significant difference was observed among groups. Only 47 % of total GLs and RCs had a score ≥50 % and none of the documents related to disorders of energy metabolism (group H) and mineral absorption and transport (group G) reached this score for this domain.

We next focused on two lysosomal storage disorders, Gaucher disease and Fabry disease, encompassed by seven and eight different GLs and RCs, respectively. Gaucher disease is an inherited disorder with an estimated birth prevalence of 1:40,000 to 1:60,000, caused by deficient activity of the lysosomal enzyme glucocerebrosidase [11, 12]. Fabry disease affects approximately 1:40,000-170,000 individuals and it is caused by a deficiency of the lysosomal hydrolase α- galactosidase A [13].

The aim was to determine whether the quality and rigour of the GLs and RCs improved over time. For this purpose, the scores referring to domains 1–6 and to the OQA domain were analyzed. For Gaucher disease, we did not find an increase in the scores referring to domains 1–6. However, a statistically significant increase in the OQA was observed (Additional file 1: Table S1 and Fig. 8a and b).

By contrast, for Fabry disease no improvement was observed in none of the appraised domains (Fig. 8c and d).