Building Bridges: Teaching Resources for Research Scientists

Building Bridges: Teaching Resources for Research Scientists

Assessment: The Hidden Variable of Achievement A Field-tested Learning Assessment Guide (FLAG) for STEM Instructors Michael Zeilik University of New Mexico (www.flaguide.org) Pop Quiz! In terms of learning gains, which instructor attribute enhances achievement the most? A. Teaching experience B. Clarity of presentations C. Energy and enthusiasm D. Deep knowledge of subject E. None of the above What Works?

20th Century Gain Results Active Learning, Mastery Learning (SD 0.5, 0.5 PSI, discussion, debates, games, role playing, controversy) Cooperative Learning (SD 0.5, 0.5 cognitive and affective; century of research, all disciplines) One-on-One Tutoring (SD = 2, with trained tutors) What Does Not Work? (Small gains SD < 0.3) EVERYTHING ELSE! Lectures (standard model), reinforce memorization! Unstructured discussion, supervised independent study, autonomous small groups, self study Audio-tutorials, programmed instruction, computer-based instruction, instructional television, Web-based instruction

What Works in Context: Physics & Astronomy Disciplinary education research, uses tools of the trade to conduct experiments Create a mostly empirical robust knowledge base about learning in physics & astronomy Probe initial state (prior knowledge), final state (learning outcomes), and student thinking Measurement: Assessment! Galileo Galilei: Measure what is measurable, and make measurable what is not so. But: Measure what you value, and value what you measure. (M. Zeilik) FLAG Features

Assessment Basics What is this assessment business all about? Making Goals What do you want to measure? Classroom Assessment Techniques (CATs) How do you measure it? Searchable DatabaseToolbox What are good tools for measurement? All peer reviewed and evidence based FLAG CATs Attitudinal Surveys: E. Seymour, E. Lewis Concept Tests: A. Ellis Concept Maps: M. Zeilik

Conceptual Diagnostic Tests: M. Zeilik Interviews: M. Smith, S. A. Southerland Performance Assessments: T. Slater Portfolios: T. Slater Scoring Rubrics: D. Ebert-May Student Assessment of Learning Gains: E. Seymour Weekly Reports: E. Etkina Mathematical Thinking: M. Swan, J. Rideway Multiple-Choice Tests: J. Parkes Minute Papers: M. Zeilik

Minute Paper Take a few minutes at the end of class and ask for a written response to: What was the most important concept you learned in class? What important question remains unanswered? What was the muddiest point of this class? Few Minute Paper: Teams reach consensus, submit written report Analysis: Sort into themes (cards) Weekly Report: Extended minute paper Concept Tests Quick feedback on conceptual (not factual) understanding Instructor gives conceptual question with choices (common misconceptions) After a minute, whole class responds (hands, flash cards, class polling system) Instructor assesses responses: If most incorrect, pair up do discuss

(peer teaching) Class response again to gauge mastery; instructor adapts in real time Earth asteroid The magnitude of the force exerted by the asteroid on the Earth is a) larger than the magnitude of the force exerted by the Earth on the asteroid b) the same as the magnitude of the force exerted by the Earth on the asteroid c) smaller than the magnitude of the force exerted by the Earth on the asteroid d) zero. (the asteroid exerts no force on the Earth) Results: Midwestern Universities (Dostal) (P111, conceptual; P221 calc; P112 algebra) Larger than: Same as:

Smaller than: Zero: N P111 P221 P112 2% 2% 5% 13% 9% 24% 62% 84% 67% 23% 101

5% 91 5% 101 Student Assessments Attitude Surveys: Perceptions about course, discipline; seem easy-not!, best based on a robust, field-tested model Minute Paper: Given limited resources, what one change would you make to improve this course?; sort by themes Student Assessment of Learning Gains (SALG): Probes learning gains that students perceive; avoids performance critiques; easily customized (15 min); available on-line UNM P102 SALG Spring 2003 (Mean = 3.38)

Error Bars are Standard Deviations 3.99 Videos 3.75 CT/CRS 4.15 Coop quizzes 3.71 Activities 3.64 Team work 3.52 Discussions

3.68 Instructor talk 1.00 2.00 3.00 4.00 Rating (scale 1 - 5) 5.00 P102 UNM Spring 2003 Gender MattersSometimes! Female Male Posttest

Confidence All p < 0.002 Pretest confidence p < 0.0001 Last math p < 0.002 Science SelfAptitude NS! Math SelfAptitude NS!

1.00 2.00 3.00 4.00 Score (1 = very poor, 5 = very good) 5.00 Cooperative Quiz Gains by Gender P102 UNM Spring 2003 Quizzes Female Effect size (F) Male Effect size (M)

1.00 0.80 0.60 0.40 0.20 0.00 Practice Quiz #1 Quiz #2 Quiz #3 Mean Does it stick? (Coop quizzes/Test) A101 UNM Fall 2000 Identical Items (p = 0.96) 120 100

80 60 40 20 Percent correct 0 Q1-12 Q2-4 Q2-9 Q2-8 Q2-7 Q3-6 Q3-7 Item Pre % Post % Test#1 Attitude Results: Intro Astro & Physics Measure (50%= neutral) UNMA101 (n = 1 31) CMUA111 ( n =224)

UNMA271 (n = 8 2) UNMP102 (n = 31) Pretest (%) Posttest Gain and (%) Effect Size 62 10 63 57 55 78

82 60 60 Not significant Not significant Not significant Not significant Conceptual Diagnostic Tests Ideally research-based on misconceptions, revealed by student think aloud interviews Measures pre/post conceptual gains as a summative assessment Force Concept Inventory (FCI; 1985);

Astronomy Diagnostic Test (ADT; 1999) version 2; national baselines; large data sample (about 5,000) Follow protocol! ADT 2 UNM Fall 2000 vs. National F) = 0.84 => 80% of postscores above mean of pres M) = 0.53 => 70% of postscores above mean of pres 30 25 30 25 20 26 28 22 20

Females 15 Males 10 All 5 0 Gain (National) (Pre = 5346; Post = 3842) Gain (UNM) Pre/post: ADT National Project/UNM = (post% - pre%)/(100% - pre%) 60 50

Standard errors plotted 38 40 30 32.4 Score (%) 20 10 53.7 47.3 56 41.5 34 27

U N M U N M 0 Pre-course (N=5346) Overall: = 0.22 Men: = 0.25 Post-course (N=3842) Women: = 0.20 UNM: = 0.31 P102 UNM Spring 2003 Pre/Post Female

Male 70% 60% 50% 40% 30% 20% Percentage 10% 0% FCI Pre% FCI Post% FCI GRC FCI+GRC Pre(FCI Post(FCI + GRC) + GRC)

Classroom Assessment: Good News! Well-done formative assessment results in a pre/post gain of about 0.5 standard deviation (70% rather than 50% on a standardized test)

Recently Viewed Presentations

  • Welcome to the Daily Sting! Yulee Hornets SOAR

    Welcome to the Daily Sting! Yulee Hornets SOAR

    Kwanzaa 2019 December 26--January 1, 2020. Created by Dr. Maulana Karenga, professor. at California State University, Long Beach in. 1966. Is an annual week-long celebration. Created to unite African-Americans after
  • Pres. Elect Trump & the 115th Congress: The

    Pres. Elect Trump & the 115th Congress: The

    Her father-in-law Richard DeVos, founder of Amway and owner of the Orlando Magic NBA team. He played a role in establishing Michigan State University's medical campus in Grand Rapids. ... PowerPoint Presentation Last modified by:
  • Welcome to 8th Grade Physical Science

    Welcome to 8th Grade Physical Science

    What is the scientific method? a series of steps that scientists use to answer questions and solve a problem. Here we identify the steps of the scientific method by IGHEAD:
  • Shout To The Lord My Jesus, my Savior

    Shout To The Lord My Jesus, my Savior

    Shout To The Lord My Jesus, my Savior Lord there is none like You All of my days, I want to praise the wonders of Your mighty love My comfort, my shelter Tower of refuge and strength Let every breath,...
  • Frontal Systems Behavior Scale™ (FrSBe™) Janet Grace, PhD ...

    Frontal Systems Behavior Scale™ (FrSBe™) Janet Grace, PhD ...

    Frontal Systems Behavior Scale™ (FrSBe™)Janet Grace, PhD, and Paul F. Malloy, PhDOverview. The FrSBe fills a gap in the assessment of frontal systems behavioral syndromes by providing a means to identify and quantify disordered natural behavior so that problems may...
  • Picking Interventions and Strategies, Part I: Problem Analysis/Mapping

    Picking Interventions and Strategies, Part I: Problem Analysis/Mapping

    I went over South Side specific data for perinatal periods of risk, which shows that the biggest contributor for South Side infant and fetal deaths are those deaths in the maternal health and prematurity box. As isn't typical, the rate...
  • &lt;Title Arial Bold 36&gt;

    <Title Arial Bold 36>

    An extension of the methodology of cumulative survival quantile curves with bootstrapped extreme value theory methods (Peng and Qi 2008; Heffernan and Tawn 2004) to provide confidence bands. Two approaches, (1) nonparametric and (2) semiparametric methods, are proposed and their...
  • What does &quot;Non Standard English&quot; mean?

    What does "Non Standard English" mean?

    What does "Non Standard English" mean? According to Wolfram Et Schilling-Estes, Vernacular or nonstandard varieties are those "varieties of a language that are not classified as standard dialects' " (305).