Correcting calculation and data errors reveals that the original conclusions were incorrect in “The best drug supplement for obesity treatment: a systematic review and network meta-analysis”

Abstract

The goal of this study was to reproduce and evaluate the reliability of the network meta-analysis performed in the article “The best drug supplement for obesity treatment: A systematic review and network meta-analysis” by Salari et al. In recent years, it has become more common to employ network meta-analysis to assess the relative efficacy of treatments often used in clinical practice. To duplicate Salari et al.‘s research, we pulled data directly from the original trials and used Cohen’s D to determine the effect size for each treatment. We reanalyzed the data since we discovered significant differences between the data we retrieved and the data given by Salari et al. We present new effect size estimates for each therapy and conclude that the prior findings were somewhat erroneous. Our findings highlight the importance of ensuring the accuracy of network meta-analyses to determine the quality and strength of existing evidence.

Introduction

In the publication by Salari et al. [1], “The best drug supplement for obesity treatment: A systematic review and network meta-analysis,” the authors used a network meta-analysis to analyze the effectiveness of several anti-obesity medications. In the history of obesity pharmacotherapy, several promising treatment candidates have arisen only to be discontinued owing to unacceptably high safety concerns [1]. The potential value of obesity pharmacotherapy encourages the publication of meta-analysis papers that combine and rigorously compare studies to determine the quality and strength of the existing evidence.

Table 1 Re-extracted information on mean age and treatment alongside data reported in the Salari et al. paper

A network meta-analysis uses both direct and indirect data from a network of trials to compare three or more treatments at once in a single study. The use of network meta-analysis to evaluate the relative efficacy of therapies often used in clinical practice has gained popularity recently [3]. There are several advantages to using network data for meta-analytical purposes, including deriving more exact estimations of the relative impact of each intervention in the network. Additionally, using network meta-analysis allows investigators to rank the interventions included in the analysis [4]. Credible inference in a network meta-analysis is premised on the assumption that the different studies included in the analysis are similar in terms of all major features that might affect the relative effects [5] and that the analysis was properly conducted. It is therefore essential to ensure that meta-analyses and network meta-analyses are conducted correctly.

Upon inspection of Salari’s et al. paper, we uncovered several data reporting and extraction errors that render the results invalid. We brought these errors to the attention of the authors, leading them to publish a correction [6], but the correction did not address all the errors that we discussed. The correction addressed the following: in Table 1, the treatment reported for the Davies et al. study [7] was incorrect (liraglutide 0.3 mg should be liraglutide 3.0 mg) and the wrong supplement was reported for the Greenway et al. study [8] (naltrexone + bupropion 16.0 mg and naltrexone + bupropion 32.0 mg; should be naltrexone 16.0 mg + bupropion and naltrexone 32.0 mg + bupropion).

Because the correction issued by the authors did not address all the errors we previously identified, we redid the network meta-analysis. Note that the analysis by Salari et al. involved 11 parallel studies which we refer to as the “original studies”: Apovian et al. [9], Aronne et al. [10], Davies et al. [7], Fidler et al. [11], Gadde et al. [12], Greenway et al. [8], Le Roux et al. [13], Lu et al. [14], O’Neil et al. [15], Pi-Sunyer et al. [16], and Smith et al. [17]. Here we report the discrepancies between what Salari et al. reported and what we obtained by extracting data directly from the original studies.

Methods

Data extraction and evaluation

We first attempted to collect the original datasets and code from the corresponding author. We reached out to the corresponding author of the original research manuscript on October 27, 2021, asking them to share their data and the R code used to generate their results. Dr. Mohammadi provided us with two materials: the appendix of the preliminary results (a Microsoft Excel file) and an R script file. Salari et al. carried out their systematic review and network meta-analysis, which we refer to as the “original research”, by conducting a systematic database search, categorizing documents for evaluation, applying inclusion and exclusion criteria, extracting data, and conducting the network meta-analysis. The data reported in the original research paper were from participants who completed post-treatment assessments.

We sought to recapitulate this analysis by extracting the same data from completers in the 11 original studies. We extracted data (sample size, mean, standard deviation, treatment name, etc.) directly from the original studies listed in Tables 1 and 2 of the Salari et al. paper. The sample size, mean, and standard deviation refer to each arm of every study. The data extracted were verified by two or more researchers.

Table 2 Re-extracted information on sex and number of participants alongside data reported in the Salari et al. paper

Analysis

We used the extracted data to calculate the effect size for each treatment using Cohen’s D metric to estimate the weight difference between groups due to changes from baseline. Upon review of the original studies, however, we found that the data in some studies were reported differently from what Salari et al. stated in their original publication¹ and in the correction [6]. Some of the original studies did not report the actual final weight value but instead reported the mean change in weight in kilograms (kg). We thus calculated the final weight values or mean change in weight for both the original studies and number reported by Salari et al. Similarly, we computed the number of participations based on the sex distribution (percentage) reported in the original studies. We then recapitulated the network meta-analysis in R studio using the data we extracted to explore the most effective drug treatment for obesity. We compared our results with the data provided by the corresponding author and the data reported by Salari et al. in their original research.

Results

In Table 1 (age and treatment), Table 2 (sex and number of participants), and Table 3 (weight and weight change), we compare the data we extracted from the original studies with those reported by Salari et al. Our new estimates of the effect sizes are included in Fig. 1.

Fig. 1

Meta-analysis study of various drug supplements used in the treatment of obesity using re-extracted values based on a random-effects model

Table 3 Re-extracted information on initial mean weight, final weight, and mean weight change

We found several discrepancies between the data we extracted and the data in the spreadsheet we received from the corresponding author. For example, their dataset did not include all the studies listed in the article as having been analyzed. Specifically, Aronne et al.’s study, Le Roux et al.’s study, and Lu et al.’s study were completely missing from the dataset but were reported in the published paper.

When we carefully reviewed the information reported in Salari et al.’s Tables 1 and 2 and compared it with the information reported in the original studies, we found several data extraction errors or discrepancies. For example, in Table 1 of the original research paper, the mean age and standard deviation are completely missing for the Davies et al. study in the interventions column [7]. However, these data were reported in the original study by Davies et al.: the mean (SD) ages were 55 (10.8) years in the liraglutide 3.0 mg group and 54.9 (10.7) years in the liraglutide 1.8 mg group. We also found data extraction errors for initial average weight and average weight change in Table 2 of the Salari et al. paper. Salari et al. did not disclose whether the variance reported (e.g.: ± 6.4 in Fidler et al.’s study) in Table 2 was a standard deviation or a standard error. Standard deviation of weight change is used to calculate the treatment effect standard error. If the authors reported the treatment effect estimates and their uncertainty using standard error for some and standard deviation for others, these effects would not be directly comparable. They should not be directly used as input for the network. We found that some of the original studies used standard deviation, and other used standard error.

We believe the aforementioned discrepancies and errors in addition to the others reported in Tables 1, 2 and 3 influenced the results of the network meta-analysis performed by Salari et al. After we calculated the differences in weight loss between each drug vs. placebo and conducted a random-effects model, we arrived at different values for the effect size (and 95% CI) compared with those reported in Fig. 6 of Salari et al. Salari et al. claimed to show the standardized mean difference (SMD). However, they reported mean differences between the groups that differed from our recapitulated results (Figs. 1 and 2) using the re-extracted values. For example, the effect size of phentermine 15.0 mg + topiramate 92.0 mg should be -8.8 [-10.72, -6.88], not − 9.10 [-10.37, -7.83], and the effect size of pramlintide should be -1.5 [-4.17, 1.17], not − 6.50 [-13.46, 0.46].

Fig. 2

The final network diagram created from the re-extracted values

Discussion and conclusion

According to the Committee on Publication Ethics’ Retraction Guidelines, retraction should be considered if there is “clear evidence that the findings are unreliable… as a result of a major error (e.g., miscalculation or experimental error)” [18]. When the errors in the paper by Salari et al. are corrected, we find a substantially different rank order of drugs in terms of the most effective weight-loss medications. We respectfully believe that Salari et al.’s network meta-analysis should be retracted because the conclusion drawn is inaccurate owing to miscalculation and inaccuracy of the data reported.

Additionally, as one reviewer noted, “It is also problematic that only completers were analyzed. A sensitivity analysis using the pattern-mixture model to model the missing participants would have elucidated whether missingness may threaten the validity of the results.” [19]. We agree. Our purpose herein was to evaluate whether the results could be reproduced (as defined by the National Academy of Sciences [20], They could not be. Future research should determine the answers obtained when the analyses are conducted optimally. Obtaining such answers will require a new full-scale endeavor including sensitivity analyses to respond to concerns around treatment of missing data.