Evaluation of State Oral Health Plans Paul W. Mattessich, Ph.D.
State Oral Health Plans • A comprehensive, integrated approach to meeting the oral health needs of a state’s population. • Specific, measurable, time-phased objectives
Importance of Planning “If you don’t know where you are going, any road will take you there.” “Failing to plan is planning to fail.”
Oral Health Plan Components • Oral health infrastructure • Healthy People 2010 objectives • Water fluoridation • School based/linked sealant programs • Strategies to identify best practices • Evaluation strategies • And others
Why evaluate state plans? • Promote effective development and implementation • Can determine what worked and did not work – then make adjustments(in current phase of work, or with next plan) • Enables states to learn from one another
Categories ofEvaluation Questions • Process: Do plan development activities occur in a way that will nurture engagement of relevant stakeholders for implementation of the plan? 2. Content: Does the plan contain the necessary elements?
3. Dissemination: Do distribution activities bring the plan to all relevant stakeholders, decision-makers? 4. Awareness: Do relevant audiences recognize and reference the plan? 5. Implementation: Do those supposed to take action actually take action?
Logic Model • Inputs: Data on Oral Health Conditions, etc. • Activities: Production and Dissemination of Plan – writing, approvals, presentations, etc. • Outputs: Decisions made; policies change; resource allocations change
Logic Model • Initial Outcomes: New and/or increased servicesEvaluation of state plan goes this far • Intermediate Outcomes: Initial changes in oral health care delivery; reduction in untreated conditions; etc. • Ultimate Outcomes: Changes in the health of the population
Questions about Process • Were the appropriate stakeholders involved in the development of the state plan? • Were the stakeholders satisfied with their level of involvement? • Do the involved stakeholders feel “ownership” of the plan?
Measurement • Participation records • Surveys (participants and non-participants)
Questions about Content • Does the plan include sufficient data describing the oral health of the state’s population? • To what extent does the plan address the Healthy People 2010 Oral Health Objectives? • Does the plan include the necessary components and sufficiently cover the necessary aspects of oral public health?
Does the plan reflect the current state of the art and science in oral public health? Is it science based? • Are goals and objectives stated in SMART (or similar) format? • Does the plan incorporate national goals as outlined in the CDC/DOH Oral Health Index?
Measurement Comparison, by objective individuals, of plan to template or guidelines, e.g. (see report): • CDC Oral Health State Plan Index • State Plan Index
Questions about Dissemination • Has an intentional strategy for dissemination of the plan been developed? • Has a strategy been implemented?
Measurement • Recording dissemination activities
Questions about Awareness • Do those who should know about the plan actually know about it? • Do those who should know about the plan (or parts of it) actually understand what it says? • Do awareness and understanding remain over time?
Measurement • Surveys • Meetings with legislators, key leaders
Questions about Implementation • Is the plan being used to guide activities and shape thinking? Is it used to guide decisions? • Do decision-makers and agencies reference the plan in their documents? • Have significant stakeholders endorsed the plan?
Have policies changed to conform to the plan? Has the allocation of resources changed to conform to the plan? • Have agencies/organizations identified by the plan for specific roles or actions carried out their assignments?
Measurement • Annual inventory of references (in documents and/or news articles) • Surveys • Annual checklist for completion of activities specified in plan