From Systematic Literature Reviews to Indirect Treatment Comparisons – Where Do We Start and How Do We Succeed?

Polly Field, MSc, DPhil, Richard White, MA, PhD, Oxford PharmaGenesis, Oxford, UK

Systematic literature reviews (SLR), meta-analyses, and their close relative ‒ indirect treatment comparisons (ITC) ‒ are becoming increasingly important for informing decisions about treatment and reimbursement. Anyone with an interest in evidence collated across more than one study will want to consider the evidence from these analyses. This includes clinicians following an evidence-based medicine approach, groups developing clinical guidelines, and health technology assessment (HTA) bodies or local payers deciding if a product should be reimbursed – or at what price.

SLRs, meta-analyses, and ITCs can be difficult to do well, and difficult to communicate well – that is, accurately, succinctly, and, above all, with relevance to the intended audience. This article introduces these analyses and shares experiences and tips for successful communication.

What Are SLRs and Why Are They Important?

The medical community is accustomed to seeing evidence hierarchies drawn as a pyramid, with weaker evidence forming the base and stronger evidence at the top. The message is clear – not all evidence is equal. Sitting at the top of some of these pyramids (a few different versions have been developed) are SLRs and meta-analyses, showing the greater weight given to these forms of evidence (see Box 1 below for evidence-based medicine resources – these describe evidence hierarchies).

So, what is an SLR? Essentially, it is the process of collecting evidence in answer to a particular research question – and this can be any question specified from the start. For example, what is the collected evidence for treatment X in patient group Y from randomized controlled trials of efficacy and safety,¹ or from observational studies of real-world effectiveness?^2,3 Or, there may be interest in the evidence for the impact of a disease on patients’ quality of life;^4,5 healthcare resource use, and cost of care;⁶ or on their outcomes – and whether there is an association between outcomes and a particular patient characteristic.⁷ Evidence is identified by searching publication databases, congress proceedings, and clinical trial registries for relevant studies (depending on the question, other sources may also be relevant). Once identified, evidence is extracted from the studies and synthesized qualitatively or quantitatively (this is meta-analysis) in answer to the original question.

But there is a need to be careful – just because the SLR collects evidence across studies, it does not mean that the combined evidence is necessarily of high quality. If the studies themselves are of low quality (“garbage in”), the body of evidence will be low quality (“garbage out”). This is reflected in a recent update to the evidence pyramid that re-casts SLR and meta-analyses as ways to interpret evidence, rather than necessarily being the pinnacle of evidence.⁸

Whichever way SLRs and meta-analyses are perceived – the best evidence, or the best way of interpreting evidence, it is generally agreed that they have an important role to play in advancing evidence-based medicine. This means that medical publication professionals should understand these methodologies and be able to communicate them effectively.

Box 1: Selection of resources and guidelines for reporting and interpreting SLRs, meta-analyses, and ITCs
BMJ EBM toolkit (evidence-based medicine resources and links)	https://bestpractice.bmj.com/info/toolkit
Campbell collaboration (evidence-based medicine resources and completed SLRs)	https://campbellcollaboration.org/
Centre for Evidence-Based Medicine (evidence-based medicine resources)	https://www.cebm.net/
Cochrane (SLR tools and completed SLRs)	https://www.cochrane.org/
EQUATOR Network, including PRISMA (reporting in SLRs and meta-analyses) and PRISMA NMA (extension for NMA)	http://www.equator-network.org/ http://www.prisma-statement.org/
GRADE working group (developing and presenting summaries of evidence)	http://www.gradeworkinggroup.org/
Trip database (tool for finding high-quality clinical research evidence)	https://www.tripdatabase.com/

BMJ, British Medical Journal; GRADE, Grading of Recommendations, Assessment, Development and Evaluations; ITC, indirect treatment comparison; NMA, network meta-analysis; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; SLR, systematic literature review

What Are Meta-analyses and ITCs and Why Are They Important?

Meta-analysis is a statistical technique to combine results across studies. The simplest and oldest form is the pairwise meta-analysis (see Box 2 below), but this has been augmented by an additional class of evidence synthesis techniques, the ITCs (see Figure 1 below).^9,10

Box 2: What is a pairwise meta-analysis?

In pairwise meta-analysis, the results from two or more independent studies are combined to address the original question – this is important as the studies need to have addressed the same research question. Studies are identified using SLR, and the results from each study are weighted (so that the evidence from a study with less variance has more weight than a study with more variance). These weighted results are used to derive a pooled estimate of the effect across studies. This has the advantage of providing greater power, based on the now larger total sample size.

Figure 1. Meta-analysis and indirect treatment comparison are methods for synthesizing results across studies, as identified by systematic literature review.

ITCs have been developed in response to a need for evidence. A lot of people – clinicians, HTA authorities, local payers, and patients – do not just want to see how the drug they are interested in compares with placebo; they want to know how its efficacy and safety compare with the drug or drugs they are currently using. Similarly, even if a study has included an active comparator, this may not be the comparator of interest for some people (perhaps because the standard of care differs by country), so there is still an evidence gap. There are not always randomized controlled trials available to answer these questions, meaning that pairwise meta-analysis cannot help. This is where ITC comes in, to help to fill this evidence gap while trials are being developed.

In these situations, statistics in the form of an ITC can be used to compare the relative efficacy of a drug across trials (the word “trials” is used here as these are most commonly randomized controlled trials). Network meta-analysis (NMA) is one of the best-established forms of ITC. In this type of analysis, trials are collected together into a network. In diagrams of these networks, the treatments are the nodes, and the connecting lines are the trials (see Figure 2 below). Each treatment is joined to at least one other treatment either directly (having been compared in the same trial) or by a common comparator arm, which is often (but not always) placebo. The relative treatment effect is estimated as the relative difference in outcome between the common comparator arm and the treatment arm. Like a pairwise meta-analysis, an NMA starts with an SLR to identify the evidence to be included. Trials cannot be included in a network if they are very different from the other trials; if the patient population differed in a clinically important aspect, it would not be acceptable to use NMA (for example, if severity of disease was markedly different). NMA methodology rests on there being no differences between trials in the distribution of effect modifying variables. This means that networks that include few trials per treatment may have imbalances between these variables, and this leads to a risk of bias in the results.

Figure 2. Evidence connections for network meta-analysis. In these examples, the white node (circle) could be placebo, connecting (“anchoring”) different treatments across multiple placebo-controlled trials.

Other types of ITC include matching-adjusted indirect comparison (MAIC) and simulated treatment comparison (STC).¹⁰ These ITCs are more recent but are gaining popularity in use as they aim to overcome some of the limitations of NMA. These techniques use individual patient data from one trial and match this to aggregated data from a second trial, to better match the profile of patients in the second trial and reduce heterogeneity between trials. Thus, two trials are required, and patient-level data must be available from one of the trials. A common comparator arm between trials allows for “anchored” indirect comparison with lower risk of bias than an “unanchored” indirect comparison (when there is no common comparator arm).

All ITCs come with limitations and, therefore, such analyses need to be developed and reported carefully. Stakeholders must have sufficient assurance about the methodology to be able to consider the results; if they do not understand the methodology and perceive it as a “black box,” then they are unlikely to trust the results.

What Are the Challenges for Medical Publication Professionals – and How Do We Meet Them?

As with all publications, the challenges that medical publication professionals need to meet are how to assure reach, engagement, and impact. Reach your audience via publication in an appropriate journal and capture their interest in the article. Convey information to this audience so that the findings are understandable and, where relevant, actionable – help them to engage with the content and see how it informs their practice or decision-making. Readers and reviewers will expect the analyses to be written accurately and succinctly, and to be relevant to their needs.

Tips for reaching the audience

SLRs, ITCs, and similar analyses are intended for a clinical and payer audience, not primarily an academic or technical audience. Hence, a clinical journal is often more appropriate than a technical journal. The objective is to gain acceptance from a journal, plus encourage people to read the article, once published.

• What is the advice for publication in a clinical journal?
  • Develop a pre-submission inquiry to ask if the editors will consider this manuscript. Focus on the clinical relevance of the analysis and the high quality of the methodology. It is difficult to assess likelihood of acceptance based only on a journal’s previous history, as the editors may well have recently published a similar analysis, but now consider this line of work complete and be less willing to consider a second similar analysis.
  • Ensure clinical experts are involved from the very start (of the analysis, not just the publication), so that the clinical relevance of the analysis is clearly apparent.
  • Be aware, though, that some analyses will not be publishable in clinical journals, especially if they are near the end of a series of similar studies or in a well-researched area. In this instance, a technical journal is the better option, especially if this ensures more rapid publication of the evidence.
  • Whichever journal is targeted, advocate for open access – reach and engagement with any article are increased markedly if it is not behind a paywall.
• How to encourage people to read the article?
  • Increase the reach of the article by including additional content, in the form of an animation, video, or infographic. This can dramatically increase the article’s reach, as readers click on the supplementary content ahead of downloading the article (see Figure 3 below). This material can also be repurposed later for internal or external training materials and communications.

Figure 3. Case study of an animation alongside a technical publication.¹¹ The journal, Rheumatology and Therapy, reported that an article describing the methodology and results of a matched-adjusted indirect comparison was their second most highly read paper of 2018. The full paper is here: https://link.springer.com/article/10.1007/s40744-018-0106-6 (buff.ly/2q2zRUp), and the associated animation is here: https://doi.org/10.6084/m9.figshare.5947795 (buff.ly/2CaxTao).

Tips for communicating systematic literature reviews well

Offer your readers help in navigating around the evidence and provide them with a meaningful commentary on the bigger picture.

• Readers may only need a summary of the methods, including search strategy (search terms and which databases or resources were searched) and criteria for study inclusion, as the methodology is relatively simple and familiar. The full information needed to be able to reproduce the SLR can be noted in a supplementary appendix. This leaves space to focus on the results of the SLR – what does the review show? Even if there is no quantitative synthesis (ie, meta-analysis), you should still develop a qualitative synthesis of the findings.
• Avoid having a list-based structure. You may start a section with list-like information (15 studies were identified, etc.), but quickly move on to explaining what these studies showed and any notable differences between studies. Provide fully referenced data tables, allowing readers to go back to individual studies as needed.
• Provide a sense of the strength of evidence – this may be based on a formal quality assessment of individual studies or across the body of evidence, or by focusing on the aspects of quality relevant to the research question (see Box 1 above for resources).
• When there is a lot of information, be creative with different graphical ways of presenting data. For example, heat maps allow quick, visual interpretation of a lot of information.

Tips for communicating ITCs and meta-analyses well

Help your readers to understand the analysis – why it was done, which inputs were used, what the results show, and what level of confidence they can place in the findings.

• Listen to the clinical authors and make sure their voices come through, so the rationale and implications of the study are clear – make sure it is not all about statistics. Make sure that you understand the technique and work closely with technical authors to resolve queries early in drafting to ensure that the results are reported accurately. Follow relevant guidelines and checklists for reporting (see Box 1 above).
• Take care to not write too much: focus on the aspects of methodology that are distinct for the study in question and do not repeat methodology that is standard (see Box 3 below). Use supplementary materials rather than including everything in the main text.
• Focus on the primary analysis, reporting other analyses as sensitivity analyses, with alternative networks of studies or alternative factors for matching populations, for example. Work with authors to agree on what is being reported at an early stage.¹²
• Do not overstate results. Depending on the data included, there will be varying levels of uncertainty around the results that must be acknowledged. Be careful with, for example, use of confidence intervals (“credible intervals” may be a better description). Even when statistical significance testing gives a value <0.05, be wary of claiming statistical significance. American Statistical Association guidelines can be useful in talking about strength of evidence rather than strict thresholds for significance.¹³

Box 3: Notes of caution on source information

Often, the source of information for analytical publications is highly technical statistics reports, not written for a clinical or less technically focused audience. –      Lots of tables of analyses that make it challenging to see what is important. –      Complex methodology that could be seen as a “black box” and could be mistrusted by non-statisticians. –      Very little reason given for why the analysis was done and what the results mean.

Conclusions

SLRs, meta-analyses, and ITCs can be complex but can provide valuable evidence and deserve to be communicated well. Medical publication professionals have an important role in communicating this evidence. Take time to learn about the techniques and become proficient in the terminology – contribute your creativity and flair for communication and help to advance evidence-based medicine!

References

1. Cornes P, Gascon P, Chan S, Hameed K, Mitchell CR, Field P, Latymer M, Arantes LH, Jr. Systematic review and meta-analysis of short- versus long-acting granulocyte colony-stimulating factors for reduction of chemotherapy-induced febrile neutropenia. Adv Ther 2018;35:1816–29.
2. Raouf S, Bertelli G, Ograbek A, Field P, Tran I. Real-world use of bevacizumab in metastatic colorectal, metastatic breast, advanced ovarian and cervical cancer: a systematic literature review. Future Oncol 2019;15:543–61.
3. Ziemssen T, Medin J, Couto CA, Mitchell CR. Multiple sclerosis in the real world: A systematic review of fingolimod as a case study. Autoimmun Rev 2017;16:355–76.
4. Coghill DR, Banaschewski T, Soutullo CA, Cottingham MG, Zuddas A. Systematic review of quality of life and functional outcomes in randomized placebo-controlled studies of medications for attention-deficit/hyperactivity disorder. Eur Child Adolesc Psychiatry 2017;26:1283–307.
5. Giles L, Freeman C, Field P, Osei G, Sörstadius E, van Haalen H. The impact of chronic kidney disease on quality of life: findings from a systematic literature review. Presented at the European Renal Association – European Dialysis and Transplant Association (ERA-EDTA), 24–27 May 2018, Copenhagen, Denmark.
6. Freeman C, Giles L, Field P, Osei G, Sörstadius E, Kartman B. The economic burden of heart failure: findings from a systematic literature review. Presented at the European Society of Cardiology (ESC) Heart Failure 2018 Congress, 26–29 May 2018, Vienna, Austria.
7. Edwards CJ, Kiely P, Arthanari S, Kiri S, Mount J, Barry J, Mitchell CR, Field P et al. Predicting disease progression and poor outcomes in patients with moderately active rheumatoid arthritis: a systematic review. Rheumatol Adv Pract 2019;3.
8. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med 2016;21:125–7.
9. Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, Boersma C, Thompson D et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health 2011;14:429–37.
10. Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. NICE DSU technical support document 18: methods for population-adjusted indirect comparisons in submissions to NICE, December 2016. Available from: https://scharr.dept.shef.ac.uk/nicedsu/wp-content/uploads/sites/7/2017/05/Population-adjustment-TSD-FINAL.pdf. (Accessed 11 April 2019). 2016.
11. Nash P, McInnes IB, Mease PJ, Thom H, Hunger M, Karabis A, Gandhi K, Mpofu S et al. Secukinumab versus adalimumab for psoriatic arthritis: comparative effectiveness up to 48 weeks using a matching-adjusted indirect comparison. Rheumatol Ther 2018;5:99–122.
12. Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, Ioannidis JP, Straus S et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med 2015;162:777–84.
13. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat 2016;70:129–33.