Spin in the abstracts of systematic reviews and metaanalyses focused on percutaneous coronary intervention.

CONTEXT
"Spin" is a form of bias that involves highlighting study results in a way that presents the conclusions about benefit or efficacy beyond the scope of the data. Spin in the abstract of published studies has the potential to affect patient care, making investigations about its presence and prevalence important for readers.


OBJECTIVES
To evaluate the most severe types of spin in the abstracts of systematic reviews and metaanalyses focused on percutaneous coronary intervention.


METHODS
Using a cross sectional study design, the authors searched MEDLINE and Embase with the terms "percutaneous coronary intervention," "percutaneous coronary revascularization," "PCI," "systematic review," "meta analysis," and "meta-analysis." To be considered for this study, the article must have (1) focused on PCI; (2) had either a systematic review or metaanalysis study design; (3) been conducted on human subjects; and (4) been available in English. Reviews were excluded if these criteria were not met. Each included article was assessed for the nine most severe types of spin as defined in a previously published article, as well as other study characteristics (type of intervention being compared, date the review was received, adherence of systematic review and/or meta-analysis to Preferred Reporting for Systematic Reviews or Metanalyses (PRISMA) guidelines, requirement of PRISMA guidelines by the publishing journal, the publishing journal's five-year impact factor, and sources of funding).


RESULTS
Our database search retrieved 7,038 records; 2,190 duplicates were removed. Initial title and abstract screening led to the exclusion of 4,367 records, and an additional 281 records were excluded during full text screening. An arbitrary limit of 200 articles was applied for this analysis; five additional articles were excluded for ineligible study design, so 195 were included in our final analysis. Spin was present in the abstracts of 43 studies from that pool (22.1%). Spin type 3-selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention-occurred most frequently (29; 14.8%). The presence of spin was not associated with any of the extracted study characteristics.


CONCLUSIONS
Our data showed that spin occurred in more than one in every five systematic reviews or metaanalyses of PCI. Spin has the potential to distort a reader's ability to translate the true findings of a study; therefore, efforts are needed to prevent spin from appearing in article summaries.

Researchers have long used systematic reviews to synthesize previously published studies for clinicians; systematic reviews "…efficiently integrate existing information [,] provide data for rational decision making, [and] establish whether scientific findings are consistent and can be generalized across populations, settings, and treatment variations, or whether findings vary significantly by particular subsets [1]." Yet, many clinicians read only the abstract rather than the full text version of each article, owing to time constraints, inability to access the full text, or the belief that the abstract accurately reflects the manuscript data [2]. For instance, a study of resident physicians' simulated clinical decision making skills based on abstract only or full-text viewing found that 53 of 77 (68.8%) read only the abstract when conducting literature searches [3]. This finding is concerning because abstracts sometimes contain biased, misleading statements or present results that are not always supported by the main text [4,5]. One form of bias is known as "spin," which is "a specific way of reporting, intentional or not, to highlight that the beneficial effect of the experimental treatment in terms of efficacy or safety is greater than that shown by the results [6]." In 2019, a study evaluating cardiovascular randomized clinical trials found spin to be present in 53 of 93 (56.9%) abstracts [7]. The authors of that study concluded that investigators often manipulate the language of an abstract compared with the results and that readers of cardiovascular research should be aware of such misleading language [7]. Additionally, previous research has shown that spin in the abstract influences clinicians' interpretation of the results; in one study of 300 clinicians asked to review randomized controlled trials related to cancer [8] -150 of whom reviewed abstracts with spin and 150 of whom reviewed abstracts without spinparticipants were "more likely to rate a treatment as beneficial despite the primary outcome being statistically nonsignificant [8]." These findings highlight the importance of eliminating spin and all forms of misleading language in scientific literature.
Given the amount of spin identified in the previous study [7] of cardiovascular trials and the fact that clinicians tend to rely on the abstract alone to make clinical decisions [2], we evaluated the presence of spin in the abstracts of previously published systematic reviews and metaanalyses investigating percutaneous coronary intervention (PCI), a leading therapeutic intervention for coronary artery disease, with over 500,000 procedures performed worldwide each year [9]. Furthermore, we explored associations between particular study characteristics and the presence of spin in abstracts.

Methods
Oversight, transparency, reproducibility, and reporting This study did not qualify as human subject research according to the U.S. Code of Federal Regulations [10]; consequently, it was not subject to Institutional Review Board oversight. A key focus of our study design was transparency, as well as reproducibility. To achieve this aim, we uploaded study materialsincluding the protocol, data analysis scripts, and extraction formsto the Open Science Framework as each was made available [11]. Additionally, various materials including analysis scripts, inclusion/exclusion criteria, search strings, and raw data were supplied to an independent author (M.H.) for analysis to ensure reproducibility. The corresponding protocols and supplemental information of those studies are also available on Open Science Framework. The reporting guidelines of this study adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [12].

Search strategy
The search strategies were produced by one of the authors, a librarian (D.W.), and included the MEDLINE (Ovid) and Embase (Ovid) databases. These strategies were designed to yield systematic reviews and metaanalyses focusing on PCI. That author (D.W.) performed the search of MEDLINE (Ovid) and Embase (Ovid) on June 2, 2020, using the terms "percutaneous coronary intervention," "percutaneous coronary revascularization," "PCI," "systematic review," "meta analysis," and "meta-analysis." These preliminary studies were uploaded to Rayyan, a systematic review screening platform. Two authors (W.T.C., P.W.) independently screened the results of this search to remove duplicate studies and those that did not align with the eligibility criteria. At the conclusion of this process, masking was removed and discrepancies between the authors were resolved by two other authors (W.A., M.V.) who served as independent third parties to arbitrate if a consensus could not be reached.

Eligibility criteria
To be considered for this study, the article must have (1) focused on PCI; (2) had either a systematic review or metaanalysis study design; (3) been conducted on human subjects; and (4) been available in English. To qualify as a systematic review or metaanalysis, the review must have adhered to the criteria defined by PRISMA [12]. As outlined in the protocol, the number of articles included in our study was restricted to 200. In order to derive a randomized sample of studies that passed the inclusion criteria upon full text screening (during which five additional studies were eliminated for ineligible study design), 195 were uploaded to the Stata statistical package (StataCorp, LLC) for randomization.

Training
Prior to study selection, two authors (W.T.C, P.W.) underwent a detailed training on systematic reviews and metaanalyses provided by Johns Hopkins University via the Coursera site [13]. Additional training took place over two days exclusively on the nine types of spin that occur in systematic reviews as defined and classified by Yavchitz et al. [6], who classified a spin types severity based on the likelihood of distorting readers' interpretation of the results. For the purpose of the study, only the nine most severe type of spin that occur in the abstract of a systematic review were chosena methodology used in previous spin studies [14][15][16][17][18][19][20]. A list of all nine types of spin can be found in Table 1. The same two authors (W.T.C., P.W.) then received the final installment of training on A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2), an online assessment tool is used to evaluate the methodological quality of systematic reviews and metaanalyses. Each of the reviews included in the study were evaluated using this tool. An outline of this training is included in the study protocol [11].

Data extraction
The same two authors (W.T.C., P.W.) performed data extraction in a masked, duplicate manner. The authors both evaluated each systematic review and metaanalysis for the nine types of spin (Table 1). Following the evaluation of spin, the methodology quality of each study was assessed using the AMSTAR-2 assessment tool. This tool consists of a checklist of methodological components of systematic reviews and metaanalyses. The available ratings include "critically low," "low," "moderate," and "high," and the interrater reliability of published studies is typically moderate to high.
Aside from the data extracted for spin evaluation, the authors took note of the following elements for each included study: (1) type of intervention being compared (surgery, pharmacologic, or nonpharmacologic); (2) the date the review was received; (3) adherence of systematic review or metaanalysis to PRISMA or PRISMA for abstracts guidelines; (4) requirements for PRISMA guidelines adherence by the publishing journal; (5) the publishing journal's five-year impact factor; and (6) the sources of funding for each review (hospital, industry, private, public, a combination of funding including industry, a combination of funding not including industry, none, not mentioned, or other). Upon completion of data extraction, the two authors were unmasked and resolved any discrepancies. Two other authors (W.A., M.V.) provided

Statistical analysis
The overall frequency of spin and its subtypes were characterized using descriptive statistics. In our protocol, the authors prespecified the possibility of a binary logistic regression and calculated a priori power analysis to determine sample size using G*Power 3.1.9.7. A previous investigation on spin in the abstracts of systematic reviews and metaanalyses focused on the treatment of acne vulgaris, suggested that spin was present in 31% of abstracts [14]. Therefore, this power analysis was based on the following assumptions and parameters: (1) 20% of PRISMA compliant systematic reviews contain spin, and 40% of non PRISMA compliant systematic reviews contain spin; (2) a Type I error rate of 0.05 (2-tailed); (3) power of 0.80; (4) and multiple coefficients of determination of 0.10, a total of 185 systematic reviews would be needed, leading us to set the limit for our study at 200 articles, as previously outlined. These analytic decisions are all documented in our protocol [11]. Stata 16.1 (StataCorp, LLC) was utilized for all statistical analyses.

General characteristics
Our database search retrieved 7,038 records, with 2,190 duplicates that were subsequently removed. Initial title and abstract screening led to the exclusion of 4,367 records, and an additional 281 records were excluded during full text screening. Due to randomization and our limit of 200 studies, 41 of the 281 records were neither included nor excluded. An additional five records were excluded during data extraction for ineligible study design. In total, 195 articles were included for data extraction. The remaining 41 (21.0%) declared that funding was not received. The average five-year impact factor for journals that published articles in our sample was 6.16 (standard deviation, 7.9). The dates that our studies were received by their respective journals ranged from 1995 to 2020. Table 2 displays data about study characteristics.

Spin in abstracts
Forty three (22.1%) articles in our sample contained spin in their abstract. Type 3 (selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention) was the most common type, identified in 29 abstracts (14.8%). With respect to spin type 4 ("Conclusion claims safety based on non statistically significant results with a wide confidence interval"), 162 (83.1%) did not discuss safety outcomes or measures in the conclusion section of their abstract. Thus, spin type 4 could only be assessed in 33 abstracts, only two (6.1%) of which were found to contain spin type 4. Spin type 1, the most severe spin type ("Conclusion contains recommendations for clinical practice not supported by the findings"), was identified in three abstracts (1.5%). The presence of spin in   (Table 3).

AMSTAR-2 appraisals
AMSTAR-2 was used to rate the methodological quality of the reviews included in our analysis; the majority received a quality score of "critically low" (126; 64.6%). The remaining studies were rated as follows: 44 (22.5%) were of "low" quality, 21 (10.7%) were of moderate quality (10.7%), and four (2.1%) were of "high" quality. There was no significant association between spin and the methodological quality of a study (Table 3). Table 2 provides all 16 AMSTAR-2 rating items as well as all responses for the 195 articles included in our analysis.

Discussion
Nearly one quarter of the 195 abstracts we reviewed of systematic reviews or metanalyses evaluating PCI contained some form of spin. This finding is significant, as spin has been shown to affect perceptions of drug efficacy; in one study, Boutron et al. [8] recruited 300 oncologists to participate in a vignette based study in which they were asked to read a clinical trial abstract and respond to a series of questions afterward. While the abstracts were identical with respect to the findingsin both cases the endpoint was statistically non significantone abstract contained spin while the other did not. Oncologists who viewed the abstract with spin rated the experimental treatment more favorably than the other group. In our analysis, the most common (29 abstracts; 14.8%) form of spin was type 3-selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention. As an example of this form of spin, consider a systematic review of PCI vs. coronary artery bypass grafting in diabetic patients with left main and/or multivessel disease [21]. The abstract states, "PCI-DES [percutaneous coronary intervention with drug-eluting stent] was associated with higher mortality compared with CABG [coronary artery bypass graft]" and reports an effect size estimate with a p-value of 0.06; however, the authors concluded that bypass grafts had a benefit over PCI in all cause mortality Table : AMSTAR- items and frequency of responses (n=).

Yes
No Partial yes ) Did the research questions and inclusion criteria for the review include the elements of PICO?  (.)  (.)  () ) Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? despite the fact that the difference did not meet predetermined statistical significance. Overemphasis on the advantages of coronary artery bypass grafts in the paper's abstract has the potential to distort clinicians' interpretation of the statistically nonsignificant results. The presence of spin has been identified in peer reviewed research across multiple fields of medicine, including anesthesiology, emergency medicine, and otolaryngology [22][23][24]. These studies have focused on the frequency of spin in randomized controlled trials with statistically nonsignificant outcomes. Results from these studies suggest prevalence estimates for spin in trial abstracts range from 23% [23] to 70% [22]. More recently, spin has been studied within the context of systematic review abstracts. Ottwell et al. [14] found that nearly one third of 36 systematic review abstracts on treatments of acne vulgaris had some sort of spin. Our pool of abstracts of systematic reviews and metaanalyses on PCI had a somewhat lower amount of spin (22.1%). However, this finding does not diminish the seriousness of the issue of spin. The absence of spin would result in better research reporting that could thereby more accurately and effectively affect clinician choices on PCI. Prevention of spin would also help to provide improved and more appropriate treatment for patients who otherwise might not receive it if a clinician is misinformed from viewing a study containing spin.
Solutions for spin reduction include a method to determine risk factors for the presence of spin. We recommend training researchers against the use of spin, prior to their first publication, which could be done in a research ethics course. Also, many institutions operate in a "publish or perish" environment that encourages researchers to publish frequently in high-impact journals in order to obtain promotions or faculty positions [25]. As a consequence, researchers may feel pressured to include spin in the reporting of their research. We hope to see the environment shift toward higher education institutions emphasizing research quality rather than the number of publications when researchers are eligible for promotion. Next, researchers should recognize that not all research presents novel data; when publications eliminate spin usage, practitioners and patients alike may be less likely to expect novel findings with every peer reviewed publication. Several prior publications have documented bias toward publication of positive results over negative results [26][27][28][29]. Publication of articles with negative/null results can help to further research in the same manner as those with positive findings. Nonsignificant results should be favored just as highly as their significant counterparts when it comes to publication.
Lastly, it is imperative that osteopathic clinicians analyze cardiovascular research with the tenets of osteopathic medicine in mind. For example, PCI can improve the outcome of the structure of the cardiovascular system affected with coronary artery disease, thus increasing the functionality of the cardiovascular system. However, as recent trials have shown, some groups of patients receiving optimal medical therapy may have similar outcomes without the cost and recovery of an invasive procedure [9,30]. Thus, osteopathic physicians should not be misled by spin and instead provide rational treatment based on an understanding of the basic principles of body unity, self regulation, and the unique circumstances of each individual patient [31].

Limitations
In reference to the limitations of our, the classification of spin is subjective and some individuals may not agree with our classifications. Second, our study had a cross sectional design and the results should not be extrapolated to systematic reviews and metaanalyses published in other journals or in other time periods. Third, although we utilized the two largest bibliographic databases, MEDLINE and Embase, for our literature search, we may not have retrieved all studies regarding systematic reviews and metaanalyses on PCI.

Conclusions
Our data showed that spin occurred in almost one in every five of 195 systematic reviews or metaanalyses of PCI included in our study. Spin has the potential to distort a reader's ability to translate the actual findings of a study, and consequently, to affect clinical decision making. Efforts are needed to prevent and eliminate spin from occurring in abstracts of systematic reviews and metaanalyses focused on PCI.
Research funding: This study was funded by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship. The funds compensated the researchers for their time. Author contributions: All authors provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; all authors drafted the article or revised it critically for important intellectual content; all authors gave final approval of the version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Competing interests: None reported.