Positive results from a randomized trial can often be enough to convince global health policymakers to invest in wide-scale implementation of a new health intervention. But incomplete and inaccurate reporting of trial data may be inflating the potential impact of some research findings, according to a new study led by Duke data scientists.
In an article published in the journal Lancet Global Health, the researchers found that more than 80% of published results from cluster randomized trials, a type of trial common in global health, did not follow international standards for data reporting, leaving them open to misinterpretation.
“Many reports provide only partial information, and the way in which the findings are reported could be interpreted as being of greater benefit than in reality,” says Elizabeth Turner, Ph.D., associate professor of biostatistics, bioinformatics and global health and the study’s lead author.
A major source of fuzziness in many trial results lies in how authors report the probability of outcomes. In randomized trials, researchers are often seeking to compare the rates at which a particular health outcome occurs in both a test and control group, such as rates of infection among vaccinated and unvaccinated trial participants. The difference between those rates is a measure called absolute risk reduction, or ARR. If 30% of the control group was infected, but only 15% of the group who received the vaccine, the ARR would be 15%, meaning that the vaccine was estimated to prevent a bad outcome for 15% of the test group.
But many studies rely on a different statistic, known as relative risk reduction, that measures how much a treatment or intervention reduces the chances of a particular outcome relative to what they would have been without that intervention. In the vaccination example, only half as many people in the test group became infected as in the control group, meaning the relative risk reduction would be 50%.
When cited alone, either measure leaves room for misinterpretation, explains John Gallis, a biostatistician with DGHI’s Research Design and Analysis Core (RDAC) who worked on the study. A treatment that cuts the risk of getting a disease in half may sound promising, but what if the overall risk of catching the disease is only 0.01%? Would it necessarily be worth scaling up a treatment with such a small absolute benefit?
“The idea behind reporting both measures is that they provide complementary information,” says Gallis.
In fact, an international panel of clinical trial methodologists and journal editors known as the Consort Group published guidelines in 2010 that call on authors to cite both absolute and relative measures when reporting findings from cluster randomized trials. But that isn’t happening, the Duke authors say. The team studied 73 articles based on cluster randomized trials published during 2017 and found only 13 papers (18%) where both absolute and relative risk figures were clearly stated in the findings. Most (63%) cited only relative risk measures, while 11% reported only absolute risk. Another 8% provided no clear measure of the estimated effect of the intervention.
The researchers also looked at trial results that broke down outcome probabilities by group, allowing them to evaluate potential overstatement of overall impact. They found the majority – 63% -- reported data in a way that could exaggerate the potential impact of the intervention being tested.
Such incomplete data can make some health interventions appear more attractive on paper than they are in practice. Alyssa Platt, a biostatistician with RDAC and another of the paper’s authors, offers the example of lottery tickets: Buying four lottery tickets instead of one technically quadruples your probability of winning the big prize, and if you read only those odds, extra lottery tickets may sound like a wise use of resources – until you consider the absolute chances of winning remain infinitesimally small.
In the context of global health, the stakes are a lot bigger than a couple of bucks at the convenience store. Overstatement of trial results can skew the cost-benefit analysis of investing in new health practices, leading health ministers to make ill-informed decisions about how to invest scarce public health resources, says Turner.
"The choices that researchers make when reporting results from randomized trials are important, because they have influence on the perceived public health impact of interventions,” she says.
Buying four lottery tickets instead of one technically quadruples your probability of winning the big prize, and if you read only those odds, extra lottery tickets may sound like a wise use of resources.
Turner doesn’t believe authors are intentionally cherry-picking risk measures to brush up the potential impact of their interventions. It’s more likely that they aren’t in a position to implement all of the complex statistical methods involved in assessing probabilities in cluster randomized trials. For that reason, Turner encourages researchers to work with units like RDAC, which assists DGHI faculty in the analysis of trial data.
Journal editors could also do more to uphold the Consort Group standards for data reporting, Turner notes. Strikingly, the researchers found that the majority of papers they evaluated were published in journals that have endorsed the Consort guidelines, suggesting a disconnect between their ideals and the day-to-day workings with researchers.
“It’s imperative that we as researchers provide a complete picture of trial data, and imperative for journal editors to require us to do so,” Turner says. “Bad or incomplete data can easily turn into a waste of a country’s health resources.”
Turner, Gallis and Platt are also part of the BERD (Biostatistics, Epidemiology and Research Design) Core in Duke’s Department of Biostatistics and Bioinformatics, which contributed 14 authors to the Lancet paper, including several graduate students. Thanks to its unique, crowd-sourcing approach to data extraction, the research involved a total of 76 group authors working in six countries.
