The Truth, the Whole Truth, or “About Half” of the Truth: AllTrials and TrialsTracker

May 30, 2017 | News and Trends

Jackie Marchington, PhD, ISMPP CMPP™, Caudex, a McCann Health Company, Oxford, UK; Adam Jacobs, MSc, PhD, Premier Research, Wokingham, UK

This article is an opinion piece based on the “Mythbuster” presentations given at the 12th Annual Meeting and 2017 European Meeting of ISMPP.

Context

Criticism of pharmaceutical industry sponsored clinical trials tends to fall into two main areas, summed up by sentences such as “About half of all randomized clinical trials are never published, and the other half is often published selectively”1 – results disclosure and selective reporting.

In the current climate of calls for improved clinical trials transparency, results disclosure, and data sharing, there are three campaigns running in the UK that are raising awareness and bringing these initiatives to wider scientific and lay audiences. These are the AllTrials campaign and TrialsTracker, which deal with disclosure rates, and the COMPare project, which deals with selective reporting.

AllTrials

AllTrials is a worthy campaign with an excellent, very simple, goal – “All trials registered: All results reported.” Can’t argue with that – its petition in the UK has collected about 90,000 signatures at the time of writing, including ours.* It was founded in 2013 and has high profile and charismatic supporters (Ben Goldacre, Sense About Science, BMJ, PLOS, Cochrane Collaboration, and more). Though launched initially in the UK, AllTrials has international support and is being led in the US by Sense About Science USA. There are TED talks about it, there are TV interviews about it. It really is a good idea and a straightforward concept to understand.

TrialsTracker and COMPare

The TrialsTracker and COMPare projects are run from the Evidence Based Medicine (EBM) Data Lab at Oxford University, UK, of which Ben Goldacre is the Director. The EBM Data Lab is part of the Nuffield Department of Primary Care Health Sciences. It “…campaign[s] for better, transparent, timely and accessible information in healthcare.” These projects, though apparently straightforward, are a little more difficult to interpret.

TrialsTracker has a very simple remit. Are all eligible trials that are registered on the world’s biggest clinical trials registry, ClinicalTrials.gov (CT.gov), disclosing their results in the CT.gov database or publishing their results in a PubMed listed journal? This group has created an automated tool that tracks CT.gov eligible studies against the study results field in the database, and looks for the CT.gov trial registration number (NCT) in PubMed. If it finds either of these things, then it considers the results disclosed. The group publishes their results on their website, and operates a social media campaign to “name and shame” organizations with low disclosure rates as assessed by the tool.

Lastly, COMPare’s raison d’être is to track “switched outcomes” in clinical trials, by which the project team means comparing the study outcomes listed in the clinical trial protocol or registry entries with the study publication to see whether the study authors reported the outcomes fully and correctly. “Outcome switching” is the new terminology for what has historically been called “selective reporting,” which has been a known problem in trials reporting for years, and which definitely requires scrutiny. However, for the purpose of this article, we’re sticking to disclosure rates, and will revisit COMPare in the future.

What Are These Campaigns Saying?

The AllTrials campaigners consistently claim that “about half of all clinical trials have never been reported,” but the evidence they cite is selective and often incorrect. In a campaign calling for transparency, this is disappointing and counterproductive. Similarly, TrialsTracker states that since 2006, 45.2% (about half?) of eligible trials registered on CT.gov are missing results. It lists all trial sponsors with 30 or more registered trials in order of non-disclosure (proportion of “undisclosed” trials) and tweets regularly (@TrialsTracker) that these organizations have failed to disclose their trial data.

So what are the problems with these “about half” claims?

Apples and Pears

From the AllTrials supporters, the main cited source of the “half of…”  zombie statistic is a meta-analysis2 – or actually an update of an earlier meta-analysis3 – which looked at various reporting biases in clinical research. This meta-analysis at no point attempted to quantify the proportion of clinical trials that were published. It also only reported on studies up to 2003, before the onset of the International Committee of Medical Journal Editors (ICMJE) requirement for trial registration (2005), the World Health Organization (WHO) trial registration policy (2006), and the Food and Drug Administration (FDA) Amendments Act requiring summary results posting (2007).

So, what is the truth about the reporting rate of registered clinical trials? The point is, that nobody really knows. There are definitely studies that report 50% disclosure rates,4,5 but recent studies typically report much higher rates.6–10 Studies on disclosure rates use different methodologies, different “populations” (congress abstracts that go on to full publications; studies registered with a particular ethics committee; trials on a particular drug class; restricted by date, therapy area, journal, etc.), and define “published” differently. Some studies only look for journal publications,10 whereas others include summary results postings.9 Disclosure rates as high as 93% have been reported recently,8 but this rate applies only to studies relating to products approved by the European Medicines Agency (EMA) in 2012–13.

In short, the studies of clinical trial reporting rates are not comparable, and it is impossible to generalize them to come up with a firm statistic. The AllTrials soundbite that “half of all clinical trials have never been reported” is not evidence-based, and does not stand up to close scrutiny.11

Figure 1. It is not possible to make fair comparisons between apples and pears. Image licensed from http:///www.tagxedo.com under a CC BY-NC-SA 3.0 license.


Cherries

With the TrialsTracker, again we ask is everything what it seems? By “undisclosed,” the TrialsTracker team means any trial registered on CT.gov that is eligible to have reported summary results on the website (ie, has been completed for a year, though the TrialsTracker methodology extended this timeframe to two years) and which has not done so, and for which the automated search cannot find the NCT number in PubMed. Nondisclosure of clinical trial results on CT.gov is a known problem,12,13 and it’s also known that only half of the trials that have posted results on CT.gov have fully published their results in a journal article.14

TrialsTracker[1] identifies 11,371 trials with no posted results out of 25,632 eligible studies (about half, in fact [47%]) on CT.gov. From then on, however, the methodological problems start. The tool depends on the results disclosure field being ticked in the database, and it ignores links to publications in the trial record or results posted on websites/registries other than CT.gov. It also depends on the NCT number being indexed in either the correct database field on PubMed or in the PubMed abstract. We know that the database tagging in PubMed is not necessarily the best, so hardly a foolproof mechanism. Also, the tool doesn’t always correctly identify the presence of the NCT number in a PubMed abstract. For example, TrialsTracker reports that 285 of 435 (65.5%) trials registered by Sanofi have not disclosed results. After downloading the dataset and looking at the first 10 “undisclosed” Sanofi trials, of the two that were not disclosed on the company website, one was published with the NCT number in the PubMed abstract (see Sanofi trial). That may be the only anomaly amongst the 11,371 trials, but it is another flaw in the tool.

The TrialsTracker methodology and top-line findings have been published in F100015 and the authors do acknowledge some of the limitations of their automated process. However, from the correspondence on that article, it is easy to see the degree of misrepresentation of the actual rate of clinical trial results disclosure. Searches relating to Sanofi and AstraZeneca’s  “undisclosed” trials are highlighted in the correspondence, and elsewhere.16 It would appear from these findings that the nondisclosure rate of registered clinical trials may be nearer 21% than 45%. Even a cursory search on a non-US pharmaceutical company such as Astellas finds a high degree of disclosure on the company website and the Japanese trial registration website (in Japanese). Just because summary results are not posted on CT.gov or PubMed, does not mean they have not been disclosed by other means.

Figure 2. Cherry picking data sources gives an incomplete picture of disclosure rates.
Image licensed from http://wwwtagxedo.com under a CC BY-NC-SA 3.0 license.

Transparency Matters

As publication professionals, much of our day job is making sure we are being transparent about our roles, accurate with the data we are presenting, and balanced in our discussion of those data. The claims made by AllTrials and TrialsTracker are made in the interests of full disclosure and transparency – that is to be applauded. Their methods, however, are not robust and, if anything, undermine the ideals they aspire to.

Disclosures

This article is the opinion of the authors and does not necessarily reflect the opinion of their employers. JMM is a current member and AJ is a former member of the Global Alliance of Publication Professionals (GAPPTeam.org).

*JMM signed the AllTrials petition as an individual, not as an employee of McCann Health. AJ signed on behalf of his former company, Dianthus Medical.

References

  1. Maund E, Tendal B, Hrobjartsson A, et al. Benefits and harms in clinical trials of duloxetine for treatment of major depressive disorder: comparison of clinical study reports, trial registries, and publications. BMJ. 2014;348(jun04 2):g3510-g3510. doi:10.1136/bmj.g3510.
  2. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14(8):iii, ix-xi, 1-193. doi:10.3310/hta14080.
  3. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess (Rockv). 2000;4(10). doi:Health Technology Assessment.
  4. Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial Publication after Registration in ClinicalTrials.Gov: A Cross-Sectional Analysis. Sim I, ed. PLoS Med. 2009;6(9):e1000144. doi:10.1371/journal.pmed.1000144.
  5. Schmucker C, Schell LK, Portalupi S, et al. Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One. 2014;9(12):e114023. doi:10.1371/journal.pone.0114023.
  6. Rawal B, Deane BR. Clinical trial transparency: an assessment of the disclosure of results of company-sponsored trials associated with new medicines approved recently in Europe. Curr Med Res Opin. 2013;(October):1-11. doi:10.1185/03007995.2013.860371.
  7. Rawal B, Deane BR. Clinical trial transparency: an assessment of the disclosure of results of company-sponsored trials associated with new medicines approved recently in Europe. Curr Med Res Opin. 2014;30(3):395-405. doi:10.1185/03007995.2013.860371.
  8. Deane BR, Sivarajah J. Clinical trial transparency update: an assessment of the disclosure of results of company-sponsored trials associated with new medicines approved in Europe in 2013. Curr Med Res Opin. 2017;33(3):473-478. doi:10.1080/03007995.2016.1263612.
  9. Bourgeois FT, Murthy S, Mandl KD. Outcome reporting among drug trials registered in ClinicalTrials.gov. Ann Intern Med. 2010;153(3):158-166. doi:10.7326/0003-4819-153-3-201008030-00006.
  10. Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM. Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ. 2012;344(jan03 1):d7292-d7292. doi:10.1136/bmj.d7292.
  11. Jacobs A. Zombie statistics on half of all clinical trials unpublished. http://www.statsguy.co.uk/zombie-statistics-on-half-of-all-clinical-trials-unpublished/. Published 2015. Accessed March 26, 2017.
  12. Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ. 2012;344(jan03 1):d7373-d7373. doi:10.1136/bmj.d7373.
  13. Anderson ML, Chiswell K, Peterson ED, Tasneem A, Topping J, Califf RM. Compliance with results reporting at ClinicalTrials.gov. N Engl J Med. 2015;372(11):1031-1039. doi:10.1056/NEJMsa1409364.
  14. Riveros C, Dechartres A, Perrodeau E, Haneef R, Boutron I, Ravaud P. Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Med. 2013;10(12):e1001566; discussion e1001566. doi:10.1371/journal.pmed.1001566.
  15. Powell-Smith A, Goldacre B, Powell-Smith A, Goldacre B. The TrialsTracker: Automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions. F1000Research. 2016;5:2629. doi:10.12688/f1000research.10010.1.
  16. Jacobs A. The Trials Tracker and post-truth politics. http://www.statsguy.co.uk/the-trials-tracker-and-post-truth-politics/. Published 2016. Accessed March 26, 2017.

[1] Since writing, the TrialsTracker team has released its first data update, so it now includes data up to April 2017. There have been no amendments to improve their methodology.

%d bloggers like this: