1 Background

In a meta-analysis, a statistic, e. g. a mean, a ratio, etc., is taken from each base paper and combined, e. g. averaged, weighted averaged, to give a better measure of some finding. (See Chen and Peace [1] and Ehm [2]) Key requirements of statistics coming from base papers used in a meta-analysis are that the statistics are independent and unbiased, Boos and Stefanski [3].

For a meta-analysis of randomized studies, these conditions are usually met; the papers are independent and randomization gives unbiased estimates. These conditions may not be met for a meta-analysis of observational studies and that is the subject of this paper.

Epidemiology exhibits a notoriously poor record of reproducibility of published findings going back at least as far as Feinstein [4], and Mayes et al. [5], in 1988, with continuing complaints: Taubes and Mann [6], Ioannidis [7], Kaplan et al. [8], Young and Deming [9], and Breslow [10, 11], to name a few. Breslow commented that “contradictory results emanating from a plethora of irreproducible observational studies have contributed to the lack of esteem with which epidemiology is regarded by many in the wider biomedical community.” Even the popular press is speaking up; Taubes [12], and Hughes [13], are two examples. See also Wikipedia [14], Replication crisis. Ominously, there may be actual misuse and/or even deliberate abuse of model fitting methods; see Glaeser [15], Young and Deming [9]. Two groups of researchers using the same observational data base found that a treatment both caused, Cardwell et al. [16], and did not cause, Green et al. [17], cancer of the esophagus.

A Nature survey reported that 90% of scientists responding said there is a serious (52%) or minor (38%) crisis in science, Baker [18]. The state of published scientific claims is sufficiently suspect that a consumer of information from such publications should start with the presumption that any claim made is as likely as not to be wrong (it will fail to replicate).

In randomized clinical trials (RCTs), very careful attention is given to the statistical analysis. A protocol, with a statistical analysis section (SAS) and a statistical analysis plan (SAP), is developed and agreed to by the interested parties, often a drug company and the Food Drug Administration (FDA), before the study starts. One of the major concerns is the control of false positive results, a biased answer. Statistical, experimental and managerial strategies (identified in the SAP) are employed to minimize the occurrence of a false positive result. Often replication of a finding is required.

Contrast a RCT with the typical nutritional observational study. Nutritional epidemiology essentially has no analysis requirements. Typically, the researcher can modify the analysis strategy as the data is examined. Multiple outcomes can be examined and multiple variables can be used as predictors. The analysis can be adjusted by putting multiple covariates into and out of the model. Seldom, if ever, is there a written protocol. For these factors (outcomes, predictors, covariates), there is no standard analysis strategy. The improvised strategy is often try-this-and-try-that. Under these circumstances the analysis is essentially exploratory.

This report asserts that many claims made on the basis of meta-analyses of nutrition or diet and certain diseases have not been proven as the underlying papers are exploratory. Many papers exist that examine the role of sugar sweetened beverages, SSBs, in the development of related chronic metabolic diseases, such as metabolic syndrome and type 2 diabetes, the question examined by Malik et al. [19]. Google Scholar (12/22/2016) returns over 1,510,000 hits for association between diet or nutrition and diabetes; 1,290,000 hits for association between diet or nutrition and Type II diabetes; and 14,500 hits for association between consumption of sugar-sweetened beverages and Type II diabetes. These studies are almost always associational studies, and of course, association is not proof of causation. Malik et al. [19], with a meta-analysis looks at sugar-sweetened beverages and the risk of metabolic syndrome and type 2 diabetes, hereafter referred to as Malik. We examine the 10 papers [20, 21, 22, 23, 24, 25, 26, 27, 28, 29] upon which the metaanalysis is based. Our thesis is that these papers are essentially exploratory and as confirmatory papers they are statistically flawed. In addition, there may be publication bias.

A major contribution of this research is to show that the analysis strategy used in the 10 base papers produce biased statistics which are unsuitable for meta-analysis.

2 Methods

A protocol and data extraction form (DEF) were developed (Appendix) and the methods therein followed.

3 Results

Across the 10 studies (Table 1), sample sizes ranged from 4,304 to 91,249, with a median of 28,897 and a total of 332,357; smallest nominal p-values ranged from 0.0001–0.001; and largest reported relative risk (RR) ranged from 1.23–5.06 with a median of 2.07. Eighty percent (80%) of the studies reported only government funding, 10% reported both government and nongovernment funding and 10% was unfunded.

The number of outcomes, predictors and covariates for each of the 10 base papers appear in Table 2. The range and median of the number of comparisons possible across the base papers are (2 12,288) and 6.5, respectively for Space A (Table 3), and (3072 117,117,952) and 196,608, respectively for Space T (Table 4). None of the 10 papers mention correcting for multiple testing or multiple modeling or adjusting for multiplicities of any kind. All papers appear to test at the 5% level.

As it is impossible to “prove a negative,” it is the responsibility of a researcher making a claim to provide strong evidence in support of the presumed positive claim. Given the multiple testing and multiple modeling, none of these papers provide strong evidence for their claims. Any claim made could easily be a false positive. Note that each of these 10 papers should be examined separately for validity of inferences. They must stand on their own before they can be considered for combining in a meta-analysis. As the statistics used in the base papers do not provide valid evidence for claim, the validity of the claim from the meta-analysis paper is questionable.

It is useful to review order statistics, their expected values and their relation to expected p-values as a function of the number of observations in a sample. If a random sample is taken from a population, and the objects are ordered from smallest to largest, the reordered objects are called order statistics. The value of the largest order statistic in the sample does not change from its value in the unordered sample, but it is a different animal. It is the largest number in the sample. The larger the sample, the larger is the expected value of the largest object (see Table 5). Consider a sample from the normal distribution with a standard deviation of one. If there are 10 objects in the sample, then the expected value of the largest object is 1.54.

Besides the expected value, the p-value for a z-test against the value zero is given. We expect the largest value in a sample of 10 to be about 0.5 standard deviations from the mean. Now look down Table 4. As N increases, the expected value of the largest order statistic increases. In a sample of 30 we expect the largest order statistic, by chance alone, to be about 2.04 standard deviations larger than the mean (of zero). The corresponding (unadjusted) p-value is 0.0495, which would be nominally statistically significant.

It is statistically fatal to consider the largest order statistic as if it were a random observation! It is common to adjust p-values when there are many questions at issue to control the false positive error rate. This table can be used to remind a researcher that the value of an object can be large by chance alone and that a p-value can be small, again by chance alone. The value of the largest order statistics will be larger, the larger the sample and the p-value will be smaller, the larger the sample.

If, after adjustment, a p-value is not statistically significant, the researcher needs to keep in mind that the corresponding experimental value is an order statistic and needs to be judged by expected values of order statistics, not as if it were a random value from the distribution in question.

The researcher can “cut” a continuous variable to create ordered groups. The low group can be used as a reference group and the other groups can be compared to the reference group. The set of groups can be tested for linear trend. In Table 6, the number of p-values displayed in each paper is given as # tests.

Example R code is as follows:

To make things concrete, suppose that there are 60 questions at issue in a study, and to simplify this discussion, assume the questions are independent of one another, then by chance alone we would expect to see a mean value of 2.31 standard deviations and a p-value of 0.027 for the largest order statistics. Without taking order statistics and multiple testing into account, we would declare statistical significance AND we would not expect the result to replicate. We would have a false positive. Taking the value of 2.31 to a meta-analysis would be totally misleading. The classic reference is Royston [30], who also quotes a well-known formula by Blom [31].

Statistics from Table 1 of Malik et al. [19], were extracted and placed in our Table 5 and Table 6. Using their Risk Ratios and Confidence Limits we computed z-tests, unadjusted and adjusted p-values. Note that in our Table 5 and Table 6, we have two rows for Nettleton, one for diabetes and one for metabolic syndrome. After Bonferroni adjustment, there are no statistically significant results, which implies that the observed risk ratios are biased.

In our Table 6, we give some more characteristics related to statistical testing. We note the number of foods in the food questionnaires used, FFQ. The number varies from a low of 61 to a high of 165. Each of these foods could be used individually or in combination as a predictor of the health effect. The number of covariates given explicitly in Table 2 of Malik et al. [19], is given as Total. Note that their counts of covariates are smaller than the number of covariates we counted (Table 4). Clearly the number of covariates mentioned in the abstract (Table 3), is an underestimate of the number of covariates in play.

It is common to group the predictors. In this case the number of groupings varies from 3 to 5 (Table 7). Using these groupings the researchers tested for a linear trend and they also tested the highest group against the lowest group (Table 7) so there were two dose response tests. We do not know if the number of groupings might have affected the results. Finally, each paper presents a large number of reported p-values, #Tests. None of the base papers reported any adjustment for multiple testing or multiple modeling.

4 Discussions

The first point to make is that the authors of the base paper were, in effect, doing exploratory analyses. The analysis search space for each paper was vast and nominal statistical significance of 5% is, at best, a screen, not confirmatory in any sense. A major multiple testing dilemma occurred in the 1990s when genomics came on line. Lander and Kruglyak [32], argued that for claims to be believable there should be multiple testing correction over all the analysis search space. None of the ten papers we examined performed any adjustment for multiple testing or multiple modeling and that appears to be usual for analysis of Food Frequency Questionnaires, FFQs.

Here is a missing insight. In real science, a hypothesis is refined, and then retested with new data on a sharper question. The protocol is written before the new data is analyzed. There is statistical error control. There is replication. We should give greater credence to the results of the new, more definitive study. If it is positive, we say the hypothesis is supported. If the new study fails, we should consider abandoning the hypothesis and spend science resources on some other problem.

If the covariates are fixed, then nutrition studies that use FFQs offer an opportunity for finding many negative findings. In FFQ studies there are approximately 60 to 130 food questions and many of these food questions are repeated from one study to another. The statistical analysis of all foods is easily accomplished with a few lines of code. A p-value plot would facilitate examination of all the questions, Schweder and Tvoll [33].

It is rather routine for a researcher not to submit negative papers as the belief is that editors are likely to reject negative papers. Informal conversations with multiple authors of published negative studies support the difficulty of getting them published. Across the board, negative studies have a more difficult time getting published. Given that negative papers are typically not published, eventually we can have serious publication bias, positive studies are accepted as they support the current paradigm and negative studies are rejected. As far as we know, observational studies used in meta-analyses are not routinely examined for multiple testing and multiple modeling bias. For more discussion of publication bias see Wikipedia [14], Publication bias.

Humans like a good story, which becomes a useful art in the writing of a scientific paper. Authors can accentuate positive papers and downplay or even omit negative papers, see Kabat [34]. It is very easy for presumptively neutral researchers to become true believers in an existing popular paradigm, especially when there is funding. Those doing nutrition and health effects research should be held to strict scientific standards: state if a study is exploratory, refine claims coming from an exploratory study for a confirmatory study, make data sets and analysis code available, etc.

Scientifically and logically, it is not possible to prove a negative so to make a public health claim, an investigator should provide strong evidence an analysis that names all the questions at issue, and fairly adjusts for multiple testing and multiple modeling. None of the claims made in these 10 papers can be considered reliable due to potential bias, and hence they are inappropriate for inclusion in a meta-analysis.

We, the science community, are not recognizing that authors are doing exploratory data analysis over and over, year after year. They look at multiple outcomes, multiple causes, any number of covariates, and any number of predictors. They try this and try that analysis and publish a paper if they get a p-value less than 0.05 where a plausible story can be made Kass et al. [35]. If they fail to find ”statistical significance,” then it appears that they simply do not publish. Those doing meta-analyses need to realize the problem to their work. Authors, editors and consumers can become true believers in a false paradigm.

Finally, the primary author of each of the 10 papers was contacted twice asking if data used in their paper were available. None of the authors provided their analysis data set. Unfortunately, it is common for authors not to provide their analysis data set. Without access to the data sets it is not possible to adjust the analysis for multiple testing and multiple modeling. From what is available in the papers and as summarized in Table 1 of Malik [19], it appears that none of the claims made in the 10 papers would be statistically significant after adjustment. The data should be made public so that the analyses can be corrected for the bias introduced by multiple testing and multiple modeling.

5 Summary

Ten papers used in the meta-analysis study by Malik et al. [19] were carefully examined with respect to the range of analysis options open to the researcher, the size of the analysis search space. The search space for each paper is large (in many cases vast) in light of all the questions possible so that testing claims at a nominal 0.05 is problematic. Meta-analysis using these papers should also be considered unreliable until the reliability of the underlying papers is assessed or confirmatory studies are run.