metaBUS: Summarizing and Visualizing One Million Findings in ...

metaBUS: Summarizing and Visualizing One Million Findings in ...

metaBUS: Summarizing and Visualizing One Million Findings in Psychology Frank Bosco Virginia Commonwealth University researchsynthesis2018.leibniz-psychology.org 1 Acknowledgements

Krista Uggerslev James Field (WVU) Kulraj Singh (Mississippi State) Fred Oswald ~30 doctoral student coders Mike Cheung (metaSEM) Wolfgang Viechtbauer (metafor) Piers Steel (former team member)

(see metabus.org/about_us) Funding: NSF, SHRM Foundation, SSHRC, CCAL 2 Outline 13:00-13:45: Presentation The problem metaBUS: Under the hood Project history

Live demonstrations Taxonomy Database structure and fields Targeted meta-analysis

Exploratory meta-analysis Sample insights Future directions 13:45-14:00: Discussion 3 How we find information on Crock-Pots

4 How we find scientific information 21 journals * 17 years = 357 volumes 357 volumes * 5 issues = 1,785 issues 1,785 issues * 6 articles = 10,710 articles (Judge et al., 2001; Psychological Bulletin)

5 How we find scientific information 6 Why not rely on electronic search? False positives

955 words in the English language contain the letter string age Missed findings Full-text electronic search not available for imagebased PDF; not reliable; abstracts are incomplete Even if perfect, one would still spend lots of time scanning articles 7

How to improve search Curate all findings with standards-based indexing/protocols Build search and summary interface(s) 8 How to improve search

Curate all findings with standards-based indexing/protocols Build search and summary interface(s) And then, imagine instant meta-analysis Over time, could become near-instant systematic review 9

What do our findings look like? Different traditions in psychology have reported findings differently (e.g., correlations vs. means differences) Many possible goals But first consider low-hanging fruit for maximum immediate ROI zero-order

effects 10 The case of applied psychology (Also captures some personality psychology, cognitive psychology, etc.) Lets look at an example

11 A typical study in applied psychology 12 What information should you expect? Five variables mentioned

10 correlations 5 means 5 standard deviations 5 reliability coefficients 13 What information is actually available in that study?

Green: Made salient in abstract Red: Not mentioned in abstract 68 of the 78 findings (87%) are unexpected based on the abstract content. (But each is a candidate for inclusion in meta-analysis) 14 Thus, a big opportunity Its great that applied psychologists report

effect sizes as they do. Many papers contain more than 100 findings. Some areas of psychology do not take advantage of the matrix format. Other areas have only a few effects per paper which even with all data collected- may be ultimately unimpressive for instant meta-analysis. 15

Across the sciences, the need for curation of findings is becoming clear Can we improve the search process? 16 Across the sciences, the need for curation of findings is becoming clear Can we collect all findings in a database?

17 Across the sciences, the need for curation of findings is becoming clear If so, could we have living reviews? 18

Could software be created to support the reproducibility and extension of metaanalyses? Bergmann, C., Tsuji, S., Piccinini, P. E., Lewis, M. L., Braginsky, M., Frank, M. C., & Cristia, A. (2017/in press). Promoting replicability in developmental research through meta-analyses: Insights from language acquisition research. Child Development. https://osf.io/preprints/bitss/ 19 3ubnc/

Could crowdsourcing be applied? 20 metaBUS: A large-scale research curation project Semi-automate the extraction of findings Manually classify each according to a variety of codes of interest Build cloud-based platforms for search and summary

21 Method: correlation extraction INPUT OUTPUT (ABBREVIATED) Approximately 1,100,000 findings as of June 2018

22 Method: correlation extraction Variable level 23 Method: correlation extraction

Effect level 24 Method: correlation extraction 25 Method: correlation extraction

26 Database fields Article-level codes DOI full reference grant funding

Variable-level codes Verbatim variable name M SD Sample size Exact sample size (Y/N) Taxonomic code Concept reversal (Y/N) Alpha reliability type (Y/N)

Reliability value Time point Response rate Data source Data pertains to Unit of analysis Data collection country Coding confidence

27 Demonstrations R Shiny application Coding sheet Taxonomy Targeted instant meta-analysis Query format

Text-based Taxonomy-based Mixed queries Exclusions Filters Verbatim R output World map

Additional analyses Save and share results Exploratory meta-analysis 28 Sample research questions 29

Sample research questions 30 Sample research questions 31

Sample research questions 32 Sample research questions 33 Sample research questions

34 Sample research questions 35 Sample research questions

36 Sample research questions 37 38 39

40 Dominance analysis for effect size (k = 271,272) Response rate; 0.30% Sample size;

0.90% Dummy: U.S. data; 1.00% Dummy: Student data; 5.80% Publication year; 12.90% Journal h-Index;

15.60% Alpha value; 63.50% 41 Dominance analysis for effect size (k = 271,272) Response rate; 0.30%

Sample size; 0.90% Dummy: U.S. data; 1.00% Dummy: Student data; 5.80% Publication year; 12.90%

Journal h-Index; 15.60% Alpha value; 63.50% 42 Dominance analysis for alpha (k = 38,255) Sampledata;

size;3.90% 0.10% Dummy: Student Journal h-Index; 11.19% Response rate; 11.79% Publication year;

60.74% Dummy: U.S. data; 12.29% 43 Correcting correctly? = rXY / sqrt(rXX*rYY) rXY(obs) = .20

rXX = .80 rYY = .60 = .29 ASSUMPTION: rXX and rYY are independent! Khler, T., Cortina, J. M., Kurtessis, J. N., & Glz, M. (2015). Are we correcting correctly? Interdependence of reliabilities in meta-analysis. Organizational Research Methods, 18(3), 355-428.

44 (Omnibus: r = .279; N = 287,709) 45 But what Im really excited about Large-scale assessment of construct redundancy 46

But what Im really excited about Large-scale assessment of construct redundancy 47 But what Im really excited about Large-scale assessment of construct redundancy

48 Other thoughts Curating effect sizes other than correlations What else would one extract for experimental studies? Within vs. between-Ss # levels of manipulated variables

49 Thank you Questions? Comments? Complaints? 50

Recently Viewed Presentations

  • Evolved Gas Analysis of Carbonaceous Chondrites in ...

    Evolved Gas Analysis of Carbonaceous Chondrites in ...

    CM Chondrites: Murray and Cold Bokkeveld. CM (carbonaceous Mighei-like) Chondrites . Type 1 CM chondrites are completed hydrated by aqueous alteration. Both Cold Bokkeveld and Murray are classified as CM2 carbonaceous chondrites. Murray was discovered in 1950 after an observed...
  • 7.3 Volumes - Oxford School District

    7.3 Volumes - Oxford School District

    Volumes with Known Cross Sections. A solid has as its base the circle x2 + y2 = 9, and all cross sections parallel to the y-axis are squares. Find the volume of the solid. Solids with Known Cross Sections. If...
  • Introduction - inst.eecs.berkeley.edu

    Introduction - inst.eecs.berkeley.edu

    Fourier Transform pairs Gaussian Box Filter * Things we can't understand without thinking in frequency * Salvador Dali "Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln", 1976 A nice set of basis This...
  • Math Three Shifts - Alaska Department of Education & Early ...

    Math Three Shifts - Alaska Department of Education & Early ...

    Extend experience with two-dimensional and three-dimensional objects to include informal explanations of circumference, area and volume formulas, apply knowledge of two-dimensional shapes to consider the shapes of cross-sections and the result of rotating a two-dimensional object about a line; These...
  • Analysis of truss-Method of Joint

    Analysis of truss-Method of Joint

    Methods of Analysis. Method of Joints. Method of Sections. The method of joints utilizes the force equations of equilibrium for each joint. Analysis normally begins at a joint where at least one force is known and not more than two...
  • Blank Jeopardy - Western Reserve Public Media

    Blank Jeopardy - Western Reserve Public Media

    17 data points What's the probability of randomly drawing a red ball from the group above? P(red) = 2/7 P (red) = red Total balls What's the probability of not selecting a red ball? ... Toppings or Pizza Toppings How...
  • Historical Geology - Lunar and Planetary Institute

    Historical Geology - Lunar and Planetary Institute

    A sharp bend in the chain indicates that the motion of the Pacific Plate abruptly changed about 43 million years ago, as it took a more westerly turn from its earlier northerly direction. Why the Pacific Plate changed direction is...
  • Motivation to Study for a PhD Degree - LU

    Motivation to Study for a PhD Degree - LU

    PhD studies open students to . new social networks, knowledge, skills, experience. Perceived as possibility for new achievements. ... Three groups of fields of study - different types of goals. Changing field of study twice - pursue mostly personal goals.