Systematic Reviews and Meta-Analyses - McGill University

Systematic Reviews and Meta-Analyses - McGill University

Systematic Reviews and Meta-Analyses e h t And Library e n a r h c Co

Ritz Kakuma, MSc (PhD Candidate) Department of Epidemiology & Biostatistics McGill University Outline Background on SR/MA Cochrane Collaboration Cochrane Library Example on Mammography Screening

for Breast Cancer Problems with Today's Medical Literature Most studies are too small Inconclusive, often conflicting results Traditional reviews are unstructured & subjective

Biased conclusions WHO CAN KEEP UP? Over 425,000 trials published to date Over 20,000 new trials published annually Help! For General Physicians to keep current:

Read 19 new articles per day which appear in medical journals 19 x 2 hrs (Critical Appraisal) = 38 hrs per day Davidoff F et al. (1995) EBM; A new journal to help doctors identify the information they need. BMJ 310:1085-86. Statistical Quality of Medical Research getting better 2002 G. WELCH, S. GABBE Statistics Usage in the Amer J Obstet Gynecol: has anything changed? 180;584-6 All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6)

vs. Jan June 1999 (Vol. 180, No. 1 to 6) Inappropriate statistics used in 31.7% (46/145) in 1994 and 9.8% (19/195) in 1999. RESULTS?? ? Problems with Standard Reviews Lack of scientific purpose (question) Undocumented methods of literature search Unstated criteria for selecting studies No methodological assessment of selected

studies Inadequate assessment of inter-study differences in results No attempt at quantitative synthesis (pooling) to take advantage of increased power Why Systematic Reviews? Help to deal with the volume of literature Help resolve conflicting results Scientific rather than subjective summarization of

literature Can reveal new evidence Identify knowledge gaps More reliable evidence with which to aid decision making Guide clinical research by providing new hypotheses The Cochrane Collaboration - origins Archie Cochrane

It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials. Pregnancy and childbirth work - 1980s Founded 1993 Canadian Cochrane Network & Centre Aims and objectives of the CC The Cochrane Collaboration is an international

organization that aims to help people make wellinformed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions CC built on 10 Principles Canadian Cochrane Network & Centre

collaboration building on the enthusiasm of individuals avoiding duplication minimizing bias keeping up to date striving for relevance promoting access

ensuring quality continuity enabling wide participation Organization of the CC Canadian Cochrane Network & Centre

Cochrane Centres (n=12) Collaborative Review groups (n=51) Fields (n=11) Networks (n=1) Methods working groups (n=10) Canadian Cochrane Network & Centre Cochrane Centres Australasian CC Brazilian CC

Italian CC Canadian CC Nordic CC Chinese CC South African CC Dutch CC UK CC

German CC US CC (Rhode Island, Iberoamerican CC Boston, San Francisco Branches) Cochrane Review Groups Canadian Cochrane Network & Centre

Acute Respiratory Infections Group Airways Group Anaesthesia Group Back Group Breast Cancer Group Colorectal Cancer Group Consumers and Communication Group Cystic Fibrosis and Genetic Disorders Group Dementia and Cognitive Improvement Group Depression, Anxiety and Neurosis Group Developmental, Psychosocial & Learning Problems Group Drugs and Alcohol Group Ear, Nose and Throat Disorders Group

Effective Practice and Organisation of Care Group Epilepsy Group Eyes and Vision Group Fertility Regulation Group Gynaecological Cancer Group Haematological Malignancies Group Heart Group Hepato-Biliary Group HIV/AIDS Group Hypertension Group Incontinence Group Infectious Diseases Group

Inflammatory Bowel Disease Group Injuries Group Lung Cancer Group Menstrual Disorders and Subfertility Group Metabolic and Endocrine Disorders Group Methodology Review Group Movement Disorders Group Multiple Sclerosis Group Musculoskeletal Group Musculoskeletal Injuries Group Neonatal Group Neuromuscular Disease Group Oral Health Group Pain, Palliative and Supportive Care

Peripheral Vascular Diseases Group Pregnancy and Childbirth Group Prostatic Diseases and Urologic Cancers Group Renal Group Schizophrenia Group Sexually Transmitted Diseases Group Skin Group Stroke Group Subfertility Group (see Menstrual Disorders) Tobacco Addiction Group Upper Gastrointestinal & Pancreatic Diseases Grp Wounds Group Cochrane Fields & Network

Canadian Cochrane Network & Centre

Cancer Network Field Child Health Field Complementary Medicine Field Health Care of Older People Field Health Promotion and Public Health Field Neurological Network Field Occupational Health Field Prehospital & Emergency Health Field Primary Health Care Field Rehabilitation and Related Therapies Field Vaccines Field Consumer Network

Cochrane Methods Groups Canadian Cochrane Network & Centre

Applicability and Recommendations Health Economics Health-Related Quality of Life Individual Patient Data Meta-Analyses Non-randomised Studies Prospective Meta-Analysis Qualitative Methods Reporting Bias Methods Screening and Diagnostic Tests Statistical Methods Cochrane Activities

Canadian Cochrane Network & Centre Produce and update systematic reviews Hand search for RCTs Medline (and others) enhancement Review methodology COCHRANE LIBRARY

The Cochrane Library Canadian Cochrane Network & Centre unique source of reliable and up-to-date information on the effects of interventions in health care To provide information and evidence to support decisions taken in health care and to inform those receiving care Published on a quarterly basis Canadian Cochrane

Network & Centre When should you use the CLIB? For questions on effectiveness What is the effectiveness of treatment x What is an effective treatment for y Is z effective in treating y Is z better than x at treating y Canadian Cochrane Network & Centre

When not to use the Cochrane Library General healthcare questions causal, prognosis, epidemiology, etc. Statistics (prevalence and incidence) Primary research other than RCTs

Guidelines Current research Canadian Cochrane Network & Centre The Cochrane Database of Systematic Reviews (CDSR) Complete Reviews full text, regularly updated systematic reviews of the effects of health care (2,170 reviews) prepared and maintained by the Collaboration Review

Groups Protocols protocols of reviews currently being prepared, incl. expected date of completion (1,500 protocols) includes background, objectives and methods sections Canadian Cochrane Network & Centre The Database of Abstracts of Reviews of Effectiveness (DARE)

prepared by the National Health Services Centre for Reviews and Dissemination at the University of York, UK. Abstracts of quality assessed systematic reviews structured abstracts assessing the quality of previously published SRs & summarizing findings (4,118 reviews) Other reviews: bibliographic details only references to published SRs NOT assessed for quality (800 reviews) Canadian Cochrane Network & Centre

The Cochrane Central Register of Controlled Trials (CENTRAL) References Reference list of ALL identified published randomized trials Latest issue 427,807 RCTs (Medline: 97,827 articles identified as RCT publication type) Canadian Cochrane Network & Centre

The Cochrane Database of Methodology Reviews (CDMR) Complete Reviews Full-text SRs of methodological studies (10 reviews) Highly structured and systematic, covering a specific and well-defined area of methodology prepared and maintained by the Cochrane Methodology Review Groups Protocol protocols of reviews currently being prepared, incl. expected date of completion (8 protocols)

includes background, objectives and methods sections Canadian Cochrane Network & Centre Cochrane Methodology Register (CMR) References published reports of empirical studies of methods used in reviews (5,968 reports) methodological studies directly relevant to conducting a review

Canadian Cochrane Network & Centre Health Technology Assessment Database (HTA) References HTA covers prevention and rehabilitation, vaccines, pharmaceuticals and devices, medical and surgical procedures and the systems within which health is protected and maintained Ongoing projects and completed publications from HTA

organizations (4,395 citations) Canadian Cochrane Network & Centre NHS Economic Evaluation Database (NHS EED) References Structured abstracts of articles reporting economic evaluations of health care interventions Bibliographic details of articles on relevant topics (i.e., burden of illness, economic methodology, reviews of

economic evaluations) N=15,041 citations Searching the CLIB Canadian Cochrane Network & Centre Some basics ALL contents of ALL records in ALL databases are searched Ignores: punctuation & numbers Boolean terms: AND, OR, NEXT, NEAR within 6 words both ways, NOT Restricting searches Options page and choose desired restrictions

At the end of the term, add: :AU Author :AB Abstract :TI Title :ME - MeSH terms :KY - Keywords Searching the CLIB Canadian Cochrane Network & Centre MeSH

Keywords drawn from MeSH thesaurus of U.S. NLM Accompanies some but not all records Organized hierarchically in trees Permuted Index an index of all words that appear in the MeSH thesaurus used to located specific MeSH terms Canadian Cochrane Network & Centre At McGill University and affiliated sites: The pathway to the CLIB is:

McGill HSL library homepage > Databases > > Cochrane Library > web (no login or password nec.) Useful Materials Canadian Cochrane Network & Centre The Cochrane Library: Self Training Guide Interpretation of Odd-ratio diagrams WWW links, HTA database Cochrane Handbook User manual Glossary, etc...

DOWNLOAD http://www.cochrane.org/resources/training.htm Quality of Cochrane Reviews Comparison of 36 Cochrane reviews with 39 paper-based journals Cochrane reviews less prone to bias (Jadad et al. 1998) Explicit reporting of eligibility criteria (35/36 vs 18/39) Assessed trial quality (36/36 vs 7/39) No language restriction (36/36 vs 32/39)

Quality of Cochrane Reviews Also not perfect (Olsen et al. 2001) 52 Cochrane reviews from 1998 18% - conclusions not backed up by evidence All overestimated effect of intervention Discordant Reviews Direction of effect Significance

Magnitude of effect Interpretation of results Numerous possible reasons Sources of discordance Different research question Target population Intervention being studied

Outcome measures Setting Study selection Search strategy Eligibility criteria Sources of discordance Data extraction Methods of measuring outcomes and endpoints Extent of human error Quality assessment method 25 scales and 9 checklists avail for assessing RCT

quality (Moher et al. 1996) Inconsistent quality depending on instrument used Additional 8 instruments (Juni et al. 1999) QUOROM (Quality of Reporting of Meta-Analyses) = most comprehensive (Moher et al. 1999, Shea et al. 2001) Sources of discordance Analysis Appropriateness of combining results Method of synthesis: Descriptive, meta-analysis Statistical methods Bayesian

Meta-regression Frequentist Interpretation of evidence Decision Algorithm for interpreting discordant reviews Jadad, Cook & Browman. A guide to interpreting discordant systematic reviews. CMAJ. 1997 May 15;156(10):1411-6. Screening for breast cancer with mammography among women aged 50-69 years

Report prepared for the Breast Cancer Screening Unit, Cancer Branch, Health Canada Ritsuko Kakuma March 2002 SRs on mammography screening for breast cancer Fletcher et al. Report of the International Workshop on Screening for Breast Cancer. J Natl Cancer Inst 1993; 85:1644-1656. Kerlikowske et al. Efficacy of screening mammography. A meta-analysis. JAMA 1995; 273:149-154. Olsen O & Gtzsche PC. Is screening for breast

cancer with mammography justifiable? Lancet 2000. Fletcher et al. review Research Question: To assess current state of knowledge about BRCA screening, identify knowledge gaps Study Sample: Women aged 40-79 years stratified into 3 groups: 4049, 50-69, 70+ Fletcher et al. review Methodological issues: Intl Workshop on screening for BRCA in Feb 1993 (by NCI) characteristics of participants?

Search strategy not reported Study selection method not described Data source: published and unpublished data provided by the w/s participants Validity of trials assessed: Randomization, Compliance, Contamination Quality and frequency of screening test

Adequacy of FU Validity of outcome assessment Generalizability of results Fletcher et al. review Results/Conclusions: 7 trials identified For age 50-69 age group, only 5 trials provided data New York, Two-County, Malmo, Edinburgh and Canada No statistical pooling of data Together, Fletcher et al. concluded that there was substantial benefit of screening on BRCA mortality reduction of 30% after 10-12 years of FU

Kerlikowske et al. review Research Question: To determine efficacy of screening in reducing BRCA mortality by age, # MM views per screen, screening interval and duration of FU Sample: Women aged 40-74 years stratified into 3 groups: 4049, 40-74, 50-74 Kerlikowske et al. review Methodology: Search: MEDLINE, manual (reference lists) and consultations with colleagues and experts

Search terms, dates covered Study Selection English language restriction, explicit inclusion criteria Not restricted to RCTs Data extraction 2 indep extractors & 3rd for resolving discrepancies Published and unpublished data used Quality assessment not reported..not done? Analysis Pooled data of all included trials using fixed effects

Kerlikowske et al. review Results / Conclusions: 9 studies (7 RCTs & 2 CCS) have data on 50-74 year age group Comparison* Relative Risks RCT vs. Case-control studies 0.77 (0.69-0.87) vs. 0.45 (0.29-0.70) No. of MM views: 2-views vs. 1-view

0.70 (0.58-0.84) vs. 0.83 (0.71-0.97) Screening interval: 12 mo. vs. 18-33 mo. 0.77 (0.59-1.0) vs. 0.77 (0.68-0.88) FU duration: 7-9 yrs vs. 10-12 yrs 0.73 (0.63-0.84) vs. 0.76 (0.67-0.87) Screening duration: 3-5 yrs vs. 8-10 yrs 0.76 (0.62-0.95) vs. 0.78 (0.67-0.90)

CBE + MM vs. MM alone 0.80 (0.66-0.98) vs. 0.76 (0.65-0.88) Study start date: Pre-1980 vs. Post-1980 0.76 (0.67-0.87) vs. 0.83 (0.63-1.10) Kerlikowske et al. review Results / Conclusions: All studies beneficial effect of screening (4 statistically significant) All trials included in the analysis (not CCSs)

MM screening reduced BRCA mortality by 27% in this age group after 7- 9 years and 24% after 10-12 years regardless of # MM views, screening interval, duration of screening or addition of CBE Olsen & Gtzsche review Research Question: To assess effect of MM screening for BRCA on mortality and morbidity Review methodological quality of trials focus on 3 most important sources of bias Randomization blind outcome assessment post-Rx exclusions

Sample: Women without previously diagnosed BRCA on women aged <50 years and 50+ years Olsen & Gtzsche review Methodology: Search: MEDLINE, common author names search, reference lists, specialized trials registers, letters, abstracts, grey literature Search terms, dates covered Study Selection: Explicit inclusion criteria (RCT / quasi-RCT on MM screening

vs. no MM screening) Two indep reviewers for study selection, quality assessment & data extraction Authors of studies contacted for missing information Quality assessment: High, Medium, Poor or Flawed Rx process, baseline comparability, post-Rx exclusions, Consistency of number of women randomized Analysis Intent-to-Treat analysis, with fixed effect unless heterogeneous Stratified by quality (excluded flawed)

Olsen & Gtzsche review Results / Conclusions: 7 trials identified, none with high quality (2 medium, 3 poor, 2 flawed) Total mortality no impact of screening after 7- and 13 yrs FU for both medium and poor quality trials BRCA mortality no beneficial effect of screening in medium quality trials Beneficial effect among POOR quality trials pooled RR=0.69, CI 0.55-0.80 at 7 years pooled RR=0.64, CI 0.54-0.76 at 13 years

Interventions for screened group higher Over-diagnosis, increased use of more aggressive treatment such as mastectomies and tumorectomies (20-30%) Decision Algorithm for interpreting discordant reviews Identified Trials Sampling Randomization: Canadian study recruited volunteers, other used population-based lists Assessment of pre-existing BRCA: pre- vs. post-Rx Trials: Methodological issues Canadian

Efficacy trial, not effectiveness (women = screened, not invited to be screened) CBE in control group may dilute effect of screening (MM+CBE vs. CBE) Imbalance in rate of advanced cancers at Rx Malm Inconsistent numbers across multiple reports missing unaccounted for Date of entry different for the 2 groups (date of Rx vs. date of invitation to screening) Contamination: 24% of controls had MM Continuation of study: not randomized, no baseline information Stockholm

Inconsistent numbers across multiple reports missing unaccounted for Non-independence of results of 2 sub-trials (2 smaller trials with overlapping controls) Imbalanced post-Rx exclusions, Contamination (Screening for controls began as early as year 3) Gteborg Information on Rx not described adequately Contamination (Screening for controls began 3rd and 6th years) No separate data on 50+ age group Trials: Methodological issues

TwoCounty Age imbalance of 5-months older in screening group (clinically signif?) Cluster Rx method not described adequately Contamination (Screening for controls began after a few years into study) New York Baseline imbalance in previous lump, menopause and education Post-Rx exclusion ID method of current BRCA differed --self-report vs. screen Outcome assessment (cause of death): 72% non-blind (differential misclassification or true effect?) Edinburgh

Inadequate Rx Baseline imbalances screened group had higher SES Change in allocation status after Rx Imbalanced post-Rx exclusions Contamination (Screening for controls began in year 10 of FU) Differential eligibility criteria and entry dates for the 2 groups Impact on effect measure Contamination & Compliance : underestimate Introduction of screening in controls Screened vs. invited to be screened Post-Rx exclusions: overestimate Identification of BRCA at baseline more complete for

screened group..first round of screening used to exclude Dropouts and losses: equally distributed? Adequacy of Randomization: overestimate Typically overestimate treatment effect by 30-40% (Schulz et al. 1995; Moher et al. 1998; Kjaergard et al. 2001) Outcome Assessment: overestimate Misclassification Determination of cause of death may be difficult in light of multiple illnesses Comparability of Trials

Same research question? Effectiveness vs. efficacy Same Target population? Appropriate study sample? Intervention? Screening modality, frequency of screening Outcome assessment Cause of death identified the same way? Analytical methods Fletcher et al.

No precise research question Inadequate description of methods used Search strategy, trial selection, data extraction Acknowledgement of difference in quality of trials but conclude based on ALL evidence Source of some data unclear Quality assessment carried out NOT synthesizing data = appropriate But evidence does not support their conclusion that BRCA mortality is reduced by 30% as a result of screening Kerlikowske et al.

Clear research question Explicit reporting of methods used No indication that quality assessment was done Some numbers extracted dont match reference cited Missing key articles for trials (I.e., original report) Despite good reporting of methods, caution needed in interpreting results because of the questionable nature of data used Olsen and Gotzsche

Clear research question Explicit reporting of methods used Strong methodology Addressed important question: Reduced BRCA mortality not necessarily equal to improved survival Did not assess baseline comparability stratified by age groups, which is how the data was analyzed Most rigorous and properly done review of the 3 Brought potential adverse effects of screening to forefront

Conclusions of assessment: All existing evidence has methodological issues No conclusive evidence exists to support MM Other issues: Accuracy of MM (false positives?) Detection of slow-growing tumors that may never develop into invasive cancers Psychological impact of positive results US and Canada continue to recommend MM screening for women in this age group Overall conclusions MA not appropriate in many cases

Must be able to identify the potential sources of bias in both the systematic review as well as the studies included in the review Cochrane reviews are not always perfect but generally better than nonCochrane reviews

Recently Viewed Presentations

  • Ugljovodonici - nasport.pmf.ni.ac.rs

    Ugljovodonici - nasport.pmf.ni.ac.rs

    Ove grupe su ugljovodonici sa jednim vodonikovim atomom manje i nazivaju se alkil-grupe * IUPAC nomenklatura Prva dva sistema nomenklature nisu mogla da obuhvate mnoga složena jedinjenja Stoga je 1892. godine u Ženevi održan sastanak 34 hemičara koji su razmatrali...
  • Fractions - Knox County Schools

    Fractions - Knox County Schools

    Steve . Wyborney. www.stevewyborney.com. We will: Examine the Geometrydomain in the TN Math Standards and analyze the content progressions. Apply content progressions to instructional tasks. Summarize new learning so we can plan effective lessons and create aligned tasks and assessments.
  • W ha tW e W Indiana ar he

    W ha tW e W Indiana ar he

    -- Ambrose Redmoon, rock band manager. in the 1960's. Oh yeah, he was a quadriplegic as well. What do we judge as so important, it trumps our fear of rejection, failure, and questioning from colleagues, ... Moodle. PBL's. Prezi. iMovie....
  • Curve Sketching #1 - Weebly

    Curve Sketching #1 - Weebly

    curve sketching #1 ap calculus november 9-10, 2016 mrs. agnew essential question what is the significance of the mean value theorem? how do you sketch curves using derivatives?
  • Engine Removal and Disassembly - Cengage

    Engine Removal and Disassembly - Cengage

    Engine Removal and Disassembly Chapter 50 Objectives Label and organize parts prior to engine removal Remove an engine from a vehicle in a safe and methodical manner Disassemble the engine following the correct procedures Keep parts organized for reassembly Inspect...
  • AN UM H G D T O N

    AN UM H G D T O N

    Health is a 'state' and is therefore dynamic and is always changing. Health can be optimal one moment, and then events such as accidents, illness, relationship breakdown and stressful incidents can change the state of health very quickly. This means...
  • STUDY of Philippians - Clover Sites

    STUDY of Philippians - Clover Sites

    The apostle paul did not write philippians in response to a crisis, as he did with galatians and colossians. Instead, he wrote to express his appreciation and affection for the Philippian believers … paul's affection for these people is clear...
  • Victorian Period - Crawford&#x27;s World

    Victorian Period - Crawford's World

    Victorian Period. Industrial Revolution - booms & depressions . Created new towns, goods, wealth, jobs for people climbing through middle class. Social & economic changes expressed in gradual political reforms. First Reform Bill in 1832 extended vote to all men...