Trial Transparency Campaigners Challenge Medical Journals Over 'Outcome Switching'

Trial transparency efforts are taking another step forward with two new ventures aimed at maximizing the reliability of clinical trial data in studies published in international medical journals and making it easier for researchers to access and manipulate information on the drugs database of the US Food and Drug Administration.

Both initiatives involve UK transparency campaigner Dr Ben Goldacre. The first of them is looking at "outcomes switching" – the practice of selectively reporting outcomes of a clinical trial to give the results a better gloss¹.

Outcome switching has come to the fore recently with revelations of frequent discrepancies between trial registry entries and published articles in leading medical journals regarding primary and non-primary outcomes.

Outcomes can of course be legitimately changed during the course of a trial, but outcome switching is often done without declaring the changes in order to report only those outcomes most flattering to the trial drug.

Last year PLOS ONE (Public Library of Science) published an article analyzing randomized controlled trials published in July to December 2013 in five "high impact" medical journals: The Lancet, the British Medical Journal, the New England Journal of Medicine, the Annals of Internal Medicine, and the Journal of the American Medical Association².

Over that period, 137 RCTs were found, of which 25 had discrepancies relating to primary outcomes, with the outcome being changed in 20 of these trials. Moreover, differences relating to non-primary outcomes were found in 87 studies, with both omission of pre-specified non-primary outcomes and the introduction of new non-primary outcomes, the article said. "Selective outcome reporting of either interesting or positive research findings is problematic, running the risk of poorly-informed treatment decisions," it concluded.

Later that year, an article in BMC Medicine reported that discrepancies between registered and published primary outcomes were "common." In 13% of the published studies it examined, a new primary outcome was introduced, while 9% registered a primary outcome that was not reported, and 5% published a primary outcome described as a secondary one in the registry³.

COMPare Initiative

Now a new initiative at the University of Oxford's Centre for Evidence-Based Medicine, COMPare (CEBM Outcome Monitoring Project), has gone further and brought the matter directly to the attention of the journals. Between October 2015 and January 2016, a team of academics, medical students and programmers, led by Goldacre, University of Oxford, systematically checked every trial published in those top five medical journals to see if the study findings were misreported.

Goldacre, who is also the founder of the AllTrials data transparency initiative, notes that trial outcomes – such as blood pressure after one year of treatment – should be pre-specified in the protocol and in a clinical trial registry. Pre-specified outcomes are useful because they are much less likely to give a false-positive result, he says.

Goldacre says that the trial report should cover all pre-specified outcomes, and where the reported outcomes are different, this must be declared in the report, along with reasons for the change and an explanation of the timing. "This ensures a fair picture of the trial results." However, outcome switching is "an extremely common problem that distorts the evidence we use to make real-world clinical decisions."

"That is why we are trying to address this problem, through the COMPare Trials project: correcting the record, holding individual trialists and journals to account, and producing worked contemporaneous examples of outcome switching, for discussion, to drive the issue forward."

Not Reported, Or Silently Added

In COMPare, each clinical trial report was compared with its protocol or registry entry. "Some trials reported their outcomes perfectly. For the others, we counted how many of the outcomes pre-specified in the protocol or registry were never reported. We also counted how many new outcomes were silently added," the researchers said.

They found that of 67 trials checked, nine were "perfect", while a total of 300 outcomes were not reported and 357 new outcomes were "silently" added. "On average, each trial reported just 62.1% of its specified outcomes. And on average, each trial silently added 5.3 new outcomes."

When discrepancies were detected, the researchers wrote letters to the journal concerned and tracked which journals published their letters, and which did not. They sent out a total of 58 letters, of which 15 were published, 10 were still unpublished after four weeks, and 32 were rejected by the editor.

Some, such as the BMJ, issued rapid corrections, while others like JAMA and the NEJM "have refused to publish any correspondence at all," Goldacre wrote on April 29. The Lancet took a mixed approach: of 19 letters submitted, eight were published, three were accepted for future publication, two were rejected, and six had been under editorial review for as much as five months.

A key issue here is who should take responsibility for misreported outcomes, Goldacre said, adding that journals have a "clear set of roles and responsibilities here," not least because many have endorsed the CONSORT [Consolidated Standards of Reporting Trials] guidelines, which say that all pre-specified outcomes should be reported.

Lancet "Somewhat Absent"

The Lancet, for example, "have somewhat absented themselves from this discussion," Goldacre said. "Our published correction letters have, in most cases, been accompanied by letters from other researchers raising methodological criticisms of the trial in question. In all cases, our published letters have been accompanied by an author’s reply. Where this reply has reiterated significant misrepresentations or misunderstandings around pre-specification of outcomes, we have written a follow-up letter to the journal to point these out."

However, he continued, each of these subsequent letters had been rejected by the Lancet editors, and the researchers' efforts to get the Lancet editors to give a view on misreported outcomes in their journal "have not yet had a reply."

This "cessation of public discussion" is problematic for two main reasons, Goldacre said. Firstly, these author replies often portray important misunderstandings on the part of trialists about selective reporting and the mechanisms to prevent it. "Open discussion on these issues would likely help to reduce the prevalence of such misreporting, which is a common source of bias in clinical trials."

Secondly, and "perhaps even more important," there is the question of who should take responsibility and monitor outcome switching. The Lancet seems to suggest that this is a matter for trialists, Goldacre said. "We think journals should take responsibility themselves, to ensure that outcomes are correctly reported in the trials they publish. This is especially so in the case of the Lancet, because they are prominent public supporters of initiatives to improve reporting standards, such as the REWARD project on research waste, and the CONSORT guidelines on trial reporting."

US Venture

Goldacre is also involved in a new venture in the US, where he has teamed up with Erick Turner, associate professor of psychiatry and pharmacology at Oregon Health & Science University and a former Food and Drug Administration reviewer, to create a use-friendly interface that is intended to make it easier for researchers to utilize data held on the FDA's Drugs@FDA website.

As explained in an article in Scrip Regulatory Affairs' sister publication, Scrip Intelligence, the database is seen as a "goldmine" of information on drugs approved by the FDA⁴. Some of the information has never been published in the medical literature, and moreover, unlike studies published in academic journals, the information in the database adheres to the outcomes and analytical methods specified in the original trial protocols.

But extracting the information is no easy task. As Turner commented, the database is "quite a bit clunky and not very user friendly." The new project, dubbed OpenTrialsFDA, uses OCR software to convert information in the documents into useable texts and formats, and places it on an indexed database where it can be mined and linked to other data. "The value of making this information more accessible is matching it on the same trials from different sources," Goldacre said.

The project also involves scientists from the Cambridge, UK-based non-profit organization Open Knowledge International (OKI). Output from the OpenTrialsFDA venture will eventually be matched to that of the OpenTrials project, a collaboration between OKI and Goldacre that is intended to locate, match and share all publicly accessible clinical trial data.

References

1. COMPare website, http://compare-trials.org/
2. Outcome Discrepancies and Selective Reporting: Impacting the Leading Journals? PLOS, May 21, 2015, http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0127495
3. Comparison of registered and published outcomes in randomized controlled trials: a systematic review, BMC Medicine, November 2015, http://download.springer.com/static/pdf/257/art%253A10.1186%252Fs12916-015-0520-3.pdf?originUrl=http%3A%2F%2Fbmcmedicine.biomedcentral.com%2Farticle%2F10.1186%2Fs12916-015-0520-3&token2=exp=1462964491~acl=%2Fstatic%2Fpdf%2F257%2Fart%25253A10.1186%25252Fs12916-015-0520-3.pdf*~hmac=480586765f7a5935e3d29c0f70dc87912cde0d864fbb6713639d4fbd1a78dffd
4. Turning FDA's Clunky Drug Database Into A Useable Goldmine, Scrip Intelligence, May 10, 2016