Resources for State Oral Health Program Evaluators January 30, 2013 B. J. Tatro, PhD ASTDD, [email protected] Cassandra Martin Frazier, MPH, CHES CDC, [email protected] 1 General Reminders This webinar will be recorded and archived on the ASTDD website Questions will be addressed after the speakers are finished. Please type your question into the chatbox that will
appear at the end of the webinar and then click on the bubble to the right of where you type your question to send it to the moderator Please respond to the polling questions at the conclusion of the webinar 2 This presentation was supported by Cooperative Agreement 5U58DP001695 from CDC, Division of Oral Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC 3
Desired Results Increase awareness of the importance of evaluation and evaluation capacity within state oral health programs Increase familiarity with the scope of public oral health evaluation topics Increase familiarity with and access to a variety of practical and useful evaluation resources relevant to public oral health 4 PART 1 Importance of Evaluation and Evaluation Capacity in Public Oral Health
5 Evaluation: A Key Component of State Oral Health Program Infrastructure Oral Health for All Policy and Programs Planning & Evaluation Capacity Surveillance Capacity Leadership & Partnerships Coalitions & Community Advocates Resources 6 Status of Evaluation in State Oral Health Programs
In 2012, ASTDD completed a report on the status and factors impacting state oral health infrastructure. The capacity to conduct evaluation was one of these. Findings included the following: According to CDC, the number of funded states reporting using evaluation to improve programs had increased from 3 to 11 of the 12 by the end of their second state grant funding cycle In a 2009 ASTDD survey, most states indicated that they had access to evaluation expertise, but 13 did not Currently there is no way to assess and track if and how non-CDC funded states are evaluating their programs, partnerships, or policies 7 Key Factors Associated with Program Success/Improvements in Oral Health
Evaluation helps build infrastructure and enhance sustainability when results are used to improve programs, increase program visibility and demonstrate program achievements. Strong programs have evidence-based goals, conduct routine evaluation and alter their programs based on evaluation results. (ASTDD, State Oral Health Infrastructure and Capacity: Reflecting on Progress and Charting the Future, 2012) 8 Evaluation Capacity Evaluation helps build infrastructure but the capacity to do evaluation must be present Evaluation requires:
Sufficient and appropriate staffing Ongoing training Access to data Access to decision makers Financial resources 9 Importance of Evaluation in Public Oral Health In the ASTDD Guidelines for State and Territorial Oral Health Programs, there are 10 Essential Public Health Services to Promote Oral Health in the US, one of which is: Evaluate effectiveness, accessibility, and quality of personal and population-based oral health promotion activities and oral health services
10 State Roles in Public Oral Health Evaluation Create a comprehensive evaluation plan Evaluate availability, utilization, appropriateness, and effectiveness of activities and services, with partners Provide evaluation consultation to partners Validate and share results with community members, stakeholders, and other partners Use results in oral health improvement and planning Leverage and allocate sufficient resources for evaluation 11 Community partnerships,
leverage resources, advocate Policies and plans Assurance Surveillance Analyze determinants and respond Inform, educate, empower
Policy Development Assessment Essential Public Health Services to Promote Oral Health in the U.S. Laws and regulations Services Workforce Evaluation Research 12 Evaluation of Other Essential
Public Oral Health Services Virtually all of the other nine Essential Services have an evaluation component! 13 Evaluation of Other Essential Public Oral Health Services Essential Service Surveillance Evaluation-related State Role/Activities Review effectiveness of surveillance systems to track oral health status and disparities Inform, educate, empower Evaluate effectiveness and quality of communication and
promotion programs Community partnerships, Evaluate partnerships and their effectiveness in creating leverage resources, oral health improvements advocate Policies and plans Determine impact of oral health policies Laws and regulations Evaluate impact of laws and regulations Services Evaluate impact of interventions and advocacy on access to services Workforce Evaluate professional development and dental public health workforce for their effect on oral health outcomes Research
Evaluate results of implementing the research agenda 14 Funder Expectations for Evaluation Federal Government: A central pillar of good government is a culture where answering such [outcome and impact] questions is a fundamental part of program design and where agencies have the capacity to use evidence to invest more in what works and less in what does not. ( Program Evaluation, in Budget of the United States Government, Fiscal Year 2011: Analytical Perspectives, 2010, 9092. Washington, DC: U.S. Government Printing Office) Foundations: The field of philanthropy is undergoing a fundamental transition toward more performance-centered and forward-looking evaluation approaches that provide foundations and their grantees with
timely information and actionable insights. (Mark Kramer, From Insight to Action: New Directions in Foundation Evaluation, April 2007) 15 CDC Structure for Evaluation Describe the program Develop logic models Develop a comprehensive evaluation plan and objectives Document what happens in the program Maintain accurate and organized records on programs Designate a person responsible for evaluation Create success stories Improve the program
Use evaluation findings to plan program improvements CDCs Framework for Program Evaluation in Public Health 16 CDC Structure for Evaluation Incorporate evaluation into the following required activities for CDC-funded states: Leadership The surveillance system The state oral health plan Partnerships and state oral health coalitions Sealant programs Policy development 17
Competencies for Public Oral Health Evaluators CDC has identified competencies for State oral health evaluators Conducting an Evaluation Conducting Yourself as an Evaluator 1. Interpersonal skills 2. Professional Practice 3. Evaluation Approach 4. Program Description 5. Evaluation Plan 6. Program Monitoring and Tracking
7. Collect, Analyze, and Interpret Data 8. Evaluation Reporting/Communication 9. Evaluation Use 18 Competencies for Public Oral Health Evaluators Competencies based on: Guiding Principles for Evaluators, American Evaluation Association CDC, Framework for Program Evaluation Stevahn L, King J, Ghere G, Minnema J. Establishing Essential Competencies for Program Evaluators.
American Journal of Evaluation, 2005; 26 (1) (43-59) Yarbrough D, Shulha L, Hopson R, and Caruthers F (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3rd ed.). Thousand Oaks, CA: Sage ASTDD Competencies for State Oral Health Programs 19 PART 2 Resources for Public Oral Health Evaluators 20 Evaluation Planning and Management Overview of Evaluation
Resource: Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide (Updated 2011) Description: Comprehensive how to guide for planning and implementing evaluation activities based on CDCs Framework for Program Evaluation in Public Health. Designed to assist evaluators in planning and implementing practical evaluations. Includes real-world examples, tips, and worksheets. Excellent tool for beginner evaluators. Resource: W.K. Kellogg Foundation Evaluation Handbook (1998) Description: A handbook that provides a broad overview of Kelloggs philosophy and expectations for evaluation and a framework for conducting evaluation projects. 21 Evaluation Planning and Management
Evaluation Project Management Resource: Managing and Implementing an Evaluation, Guidelines and Tools for Evaluation Managers (2008, Catholic Relief Services) Description: A guidance document on evaluation management, including management skills, communications, and dealing with the unexpected. Resource: Evaluation Plans and Operations Checklist (1999, Daniel Stufflebeam) Description: A checklist for conducting preliminary meta-evaluations of the evaluation plan or process in order to strengthen the evaluation operations. Resource: Evaluation Budget Development Checklist (2001, Jerry Horn) Description: A checklist designed to assist evaluators and others to think through the development of an evaluation budget. This instrument is divided into several categories of typical cost items for conducting evaluations. 22 Evaluation Planning and
Management Evaluation Planning Resource: Community-Based Oral Health Programs: A Need and Plan for Evaluation (2005, W.K. Kellogg Foundation) Description: A report describing the major components that should be included in evaluations of community-based oral health programs, along with suggestions for specific measures that could be considered. Resource: Developing an Effective Evaluation Plan (2011, CDC) Description: A workbook to help public health program managers, administrators, and evaluators develop a joint understanding of what constitutes an evaluation plan, why it is important, and how to develop an effective evaluation plan in the context of the planning process. 23 Evaluation of Prevention and
Promotion Programs School-based Sealant Programs Resource: Seal America: The Prevention Invention, Step 10, Program Evaluation (Updated 2011) Description: This manual is designed to assist health professionals in initiating, implementing, and evaluating a school-based dental sealant program to help prevent dental caries in children and adolescents. It includes practical suggestions for evaluating program effectiveness and efficiency. Resource: Sealant Efficiency Assessment for Locals and States (SEALS) (Updated 2009, CDC) Description: An Excel-based program that captures, stores, and analyzes data from school-based sealant programs. SEALS reports generated at the school, local, and state levels contain a variety of performance measures that can be used to assess program effectiveness and economic impact.
24 Evaluation of Prevention and Promotion Programs Fluoridation Program Management Resource: Community Water Fluoridation Program Review (2004, CDC) Description: A questionnaire to assist state programs in conducting a self-assessment of their community water fluoridation program. Based on the Engineering and Administrative Recommendations for Water Fluoridation guidelines and the ASTDD Best Practices guidelines on fluoridation. To request a copy of the questionnaire, please contact Cassandra Frazier or Kip Duchon. Resource: Oral Health Maps (CDC) Description: An interactive mapping application that presents indicators of oral health and preventive interventions for oral health. Can view data by state. Resource: Fluoride Legislative User Information Database (FLUID) Description: A comprehensive database of federal and state policies
regarding community water fluoridation. Enables users to access information based on legal fact and compare their current or proposed policies with others from across the country. 25 Evaluation of Prevention and Promotion Programs Communications Resource: Communications Plan Template for State Oral Health Programs (2010, ASTDD) Description: This tool is designed to help stakeholders plan and evaluate health promotion and other state oral health communications. 26
Evaluation of Policy Work Policy Work Resource: What Gets Measured, Gets Changed: Evaluating Law and Policy for Maximum Impact. Jamie F. Chriqui, Jean OConnor, and Frank J. Chaloupka, in Using Law, Policy, and Research to Improve the Publics Health, Spring 2011 Description: An article that describes the need for and relevance of conducting systematic policy evaluation, factors to consider when evaluating laws and policies, and examples of how policy impact studies have informed policy discussions and decision making. Provides samples of evaluation questions. Resource: The Evaluation Exchange, Vol. XIII, No. 1 (Spring 2007, Harvard Family Research Project) Description: This volume is dedicated to evaluation of advocacy and policy change and has several relevant articles. Resource: The Policy Tool Guidebook (CDHP and CDC, 2009) The Guidebook includes a section on evaluation, including an assessment
of baseline capacity for use by state oral health directors or other leaders and a participant survey for those engaged in the policy process. 27 State Plan Evaluation State Plans Resource: CDC State Oral Health Plan Evaluation Workbook (~2013, CDC) Description: This is a comprehensive and practical guide to evaluating the implementation and impact of state oral health plans. Provides at-a-glance summaries, tips, and strategies for keeping stakeholders engaged, evaluation approaches and tools, resources, and action steps. Release expected in 2013. Resource: Oral Health State Plan Review Index (2003, CDC) Description: A checklist to help plan and develop the state oral health plan. Can be used to conduct a review of the existing oral health plan. Resource: State Oral Health Plan Comparison Tool (2012, CDHP)
Description: A database that provides an overview of the current state of state oral health plans and facilitates cross-state comparisons. 28 Evaluation of Surveillance Surveillance Resource: Updated Guidelines for Evaluating Public Health Surveillance Systems MMWR, July 27, 2001; 50(RR13) (1-35) Description: This report provides guidelines for evaluating surveillance systems based on CDCs Framework for Program Evaluation in Public Health, research and discussion of concerns related to public health surveillance systems, and public comments. 29 Evaluation of Leadership and
Leadership Development Leadership Development Resource: ASTDD Competencies for State Oral Health Programs (2009, ASTDD) Description: Lists competencies in seven domains, including Domain 2, Plan and Evaluate Programs. For each competency, provides four levels from basic to advanced. There are links on the ASTDD website to associated tools and information about the availability of technical assistance. Resource: The National Public Health Leadership Institute: Evaluation of a Team-Based Approach to Developing Collaborative Public Health Leaders , American Journal of Public Health, April 2005, Vol. 95, No. 4 (641-644) Description: Describes an evaluation of the National Public Health Leadership Institute, which aims to develop collaborative leaders and strengthen networks of leaders who share knowledge and jointly address public health problems. Results are based on analysis of interviews conducted 12-18 months after the program end and project final reports.
30 Evaluation of Partnerships and Collaboration Partnerships and Collaborations Resource: Handbook on Planning, Evaluating and Improving Interagency Collaboration in State Or al Health Programs (ASTDD, 2012) Description: The handbook and companion workbook provide a framework for evaluating collaboration between state oral health programs and other entities. A variety of tools, as well as oral health examples, are provided. Technical assistance is available through ASTDD. Resource: Evaluating Partnerships to Prevent and Manage Chronic Disease , FD Butterfoss, Preventing Chronic Disease, April 2009, Vol. 6, No. 2
Description: This article describes how to develop a comprehensive evaluation strategy based on partnership theory, select outcomes indicators, choose methods and tools, and use results for accountability and improving effectiveness. 31 Where to Find New Resources ASTDD website (slides from this webinar will also be posted on the ASTDD website) CDC evaluation website Join the American Evaluation Association (AEA) or a local affiliate AEA Linked-In University of Wisconsin Extension Center evaluation website Western Michigan University, the Evaluation Center Online Evaluation Resource Library National Science Foundation
website Bureau of Justice Assistance evaluation website Conferences, including the AEA Annual Conference, National Oral Health Conference and preconference workshops, and CDC/AEA Summer Institute Online and live trainings, including AEA, CDC, and colleges/universities 32 Where to Find New Resources ASTDD website (slides from this webinar will also be posted on the ASTDD website) CDC evaluation website Join the American Evaluation Association (AEA) or a local affiliate AEA Linked-In University of Wisconsin Extension Center evaluation website Western Michigan University, the Evaluation Center
Online Evaluation Resource Library National Science Foundation website Bureau of Justice Assistance evaluation website Conferences, including the AEA Annual Conference, National Oral Health Conference and preconference workshops, and CDC/AEA Summer Institute Online and live trainings, including AEA, CDC, and colleges/universities 33 Next Steps Are you interested in: Future evaluation webinars? NOHC evaluation roundtables? NOHC evaluation presentations? NOHC evaluation pre-sessions? A evaluation community of practice? An evaluation blog?
Are you willing to post evaluation articles, data sources, tools, or other resources to a shared site, if it were available? 34
Chief Executive, GTCS ... (mandatory, comprising the SPR and the SFR) The Standard for Career-long Professional Learning. The Standards for Leadership and Management (for middle leaders and Head Teachers) Key purposes of Professional Update.
Times New Roman Symbol Default Design The Spinal Cord & Spinal Nerves Spinal Cord Protection Structures Covering the Spinal Cord External Anatomy of Spinal Cord Inferior End of Spinal Cord Spinal Cord & Spinal Nerves Spinal tap or Lumbar Puncture...
79M Septic shock/faecal peritonitis. Prolonged stays in bed - developing HAP and pleural effusions. Not encouraged to be independent or self motivated.Poor nutritional intake - slow recovery and motivationLabelled Vascular Dementia as failed MMT - inappropriate timingNo rehab beds ...
Heat Cramps. Heat (muscle) cramps tend to occur later in activity with muscle fatigue and after fluid and electrolyte imbalance and increased. Dehydration, diet poor in minerals, and large losses of sodium and other electrolytes increase the risk of severe...
God would not intervene in the world by causing a few random miracles. A God who cures a man of cancer at Lourdes but doesn't stop millions starving in Africa isn't worthy of worship. A God who acts selectively isn't...