metaBUS: Summarizing and Visualizing One Million Findings in ...
metaBUS: Summarizing and Visualizing One Million Findings in Psychology Frank Bosco Virginia Commonwealth University researchsynthesis2018.leibniz-psychology.org 1 Acknowledgements
Krista Uggerslev James Field (WVU) Kulraj Singh (Mississippi State) Fred Oswald ~30 doctoral student coders Mike Cheung (metaSEM) Wolfgang Viechtbauer (metafor) Piers Steel (former team member)
(see metabus.org/about_us) Funding: NSF, SHRM Foundation, SSHRC, CCAL 2 Outline 13:00-13:45: Presentation The problem metaBUS: Under the hood Project history
Live demonstrations Taxonomy Database structure and fields Targeted meta-analysis
Exploratory meta-analysis Sample insights Future directions 13:45-14:00: Discussion 3 How we find information on Crock-Pots
4 How we find scientific information 21 journals * 17 years = 357 volumes 357 volumes * 5 issues = 1,785 issues 1,785 issues * 6 articles = 10,710 articles (Judge et al., 2001; Psychological Bulletin)
5 How we find scientific information 6 Why not rely on electronic search? False positives
955 words in the English language contain the letter string age Missed findings Full-text electronic search not available for imagebased PDF; not reliable; abstracts are incomplete Even if perfect, one would still spend lots of time scanning articles 7
How to improve search Curate all findings with standards-based indexing/protocols Build search and summary interface(s) 8 How to improve search
Curate all findings with standards-based indexing/protocols Build search and summary interface(s) And then, imagine instant meta-analysis Over time, could become near-instant systematic review 9
What do our findings look like? Different traditions in psychology have reported findings differently (e.g., correlations vs. means differences) Many possible goals But first consider low-hanging fruit for maximum immediate ROI zero-order
effects 10 The case of applied psychology (Also captures some personality psychology, cognitive psychology, etc.) Lets look at an example
11 A typical study in applied psychology 12 What information should you expect? Five variables mentioned
10 correlations 5 means 5 standard deviations 5 reliability coefficients 13 What information is actually available in that study?
Green: Made salient in abstract Red: Not mentioned in abstract 68 of the 78 findings (87%) are unexpected based on the abstract content. (But each is a candidate for inclusion in meta-analysis) 14 Thus, a big opportunity Its great that applied psychologists report
effect sizes as they do. Many papers contain more than 100 findings. Some areas of psychology do not take advantage of the matrix format. Other areas have only a few effects per paper which even with all data collected- may be ultimately unimpressive for instant meta-analysis. 15
Across the sciences, the need for curation of findings is becoming clear Can we improve the search process? 16 Across the sciences, the need for curation of findings is becoming clear Can we collect all findings in a database?
17 Across the sciences, the need for curation of findings is becoming clear If so, could we have living reviews? 18
Could software be created to support the reproducibility and extension of metaanalyses? Bergmann, C., Tsuji, S., Piccinini, P. E., Lewis, M. L., Braginsky, M., Frank, M. C., & Cristia, A. (2017/in press). Promoting replicability in developmental research through meta-analyses: Insights from language acquisition research. Child Development. https://osf.io/preprints/bitss/ 19 3ubnc/
Could crowdsourcing be applied? 20 metaBUS: A large-scale research curation project Semi-automate the extraction of findings Manually classify each according to a variety of codes of interest Build cloud-based platforms for search and summary
21 Method: correlation extraction INPUT OUTPUT (ABBREVIATED) Approximately 1,100,000 findings as of June 2018
rXX = .80 rYY = .60 = .29 ASSUMPTION: rXX and rYY are independent! Khler, T., Cortina, J. M., Kurtessis, J. N., & Glz, M. (2015). Are we correcting correctly? Interdependence of reliabilities in meta-analysis. Organizational Research Methods, 18(3), 355-428.
44 (Omnibus: r = .279; N = 287,709) 45 But what Im really excited about Large-scale assessment of construct redundancy 46
But what Im really excited about Large-scale assessment of construct redundancy 47 But what Im really excited about Large-scale assessment of construct redundancy
48 Other thoughts Curating effect sizes other than correlations What else would one extract for experimental studies? Within vs. between-Ss # levels of manipulated variables
CM Chondrites: Murray and Cold Bokkeveld. CM (carbonaceous Mighei-like) Chondrites . Type 1 CM chondrites are completed hydrated by aqueous alteration. Both Cold Bokkeveld and Murray are classified as CM2 carbonaceous chondrites. Murray was discovered in 1950 after an observed...
Volumes with Known Cross Sections. A solid has as its base the circle x2 + y2 = 9, and all cross sections parallel to the y-axis are squares. Find the volume of the solid. Solids with Known Cross Sections. If...
Fourier Transform pairs Gaussian Box Filter * Things we can't understand without thinking in frequency * Salvador Dali "Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln", 1976 A nice set of basis This...
Extend experience with two-dimensional and three-dimensional objects to include informal explanations of circumference, area and volume formulas, apply knowledge of two-dimensional shapes to consider the shapes of cross-sections and the result of rotating a two-dimensional object about a line; These...
Methods of Analysis. Method of Joints. Method of Sections. The method of joints utilizes the force equations of equilibrium for each joint. Analysis normally begins at a joint where at least one force is known and not more than two...
17 data points What's the probability of randomly drawing a red ball from the group above? P(red) = 2/7 P (red) = red Total balls What's the probability of not selecting a red ball? ... Toppings or Pizza Toppings How...
A sharp bend in the chain indicates that the motion of the Pacific Plate abruptly changed about 43 million years ago, as it took a more westerly turn from its earlier northerly direction. Why the Pacific Plate changed direction is...
PhD studies open students to . new social networks, knowledge, skills, experience. Perceived as possibility for new achievements. ... Three groups of fields of study - different types of goals. Changing field of study twice - pursue mostly personal goals.
Ready to download the document? Go ahead and hit continue!