Evaluation Realities: What the Theories and Standards Do Not Say

Les Bolt, Ph.D. Appalachian State University 2009 National Evaluation Institute Louisville, KY October 10, 2009 Evaluation Realities:What the Theories and Standards Do Not Say

Researcher Context • Ph.D. in Research and Evaluation from UVA • 30+ Years Experience Doing Formal Evaluation • 75+ External Third-Party Evaluations • 1-4 Year Terms • Local/Regional/State/National Scope • $5,000 - $15,000,000 Budget Range • Public/Foundation/Non-Profit/Private Monies • Department/College/University Level Experience in Personnel and Program Evaluation, and Student Assessment

Research Methodology • Narrative Review • Evaluation Narratives • Evaluator’s Notes • Client Feedback • Auto-Ethnographic Reflection • Evaluator Interviews

Major Hurdles That Emerged • Insufficient Clarity of Outcomes • Lack of Baseline Data and Data Sources • Conflicting Stakeholder Expectations • Inertia from Previous Program/Practices • Fluid Leadership

Insufficient Clarity of Outcomes • Process Focus Instead of Outcome Focus • Fuzzy Outcomes • Processes and Outcomes Not Logically Related

Lack of Baseline Data and Data Sources • No Reliable Pre-Measures • Pre-Measures not Related to Outcomes • No Data Available to Construct Baseline

Conflicting Stakeholder Expectations • Not All Stakeholders Involved in Project/Program Conception • Stakeholder Drop-In / Drop-Out • Expectations not Expressed at Beginning of Project

Inertia from Previous Programs/Practices • Laundry List Effect • Processes and Materials Not Clearly Defined • “Resentful Demoralization” Surfaces

Fluid Leadership • Concept People Effect (Getting $$ vs. Operating Project) • Shifting Leadership • Loss of Leadership Interest

Managing the Hurdles • Recognize • Assess Impact • Accommodate • Acknowledge Limitations

Strategies for Accommodating Hurdles • Conduct Environmental Scan before Beginning • Triangulation of Expectations • Baselines Based on Best Practices Research • Establishing Firewalls between Previous and Current Practice • Identification of Specific Changes in Governance, Staffing and Funding

Conduct Environmental Scan before Beginning • Stated Processes vs. Actual Practice • Stated Processes vs. Resource Allocation • Stated Outcomes vs. Data Collected • Rationale for Discrepancies

Triangulation of Expectations • Identification of “Real” Stakeholders • Communication of Expectations • Consensus on Expectations • Consensus on Data Needed

Baselines Based on Best Practices • Understand the Project/Program Limitations • Review Literature that Applies • Plan and Budget for Best Practices Research • Do Your Homework

Establishing Firewalls between Previous and Current Practice • Set Clear Project Dates • Be Clear with Definitions • Use Qualifying Prompts in Surveys, Questionnaires, etc. • Identify Sources of Bias

Specific Changes in Governance, Staffing and Funding • Look for Explicit Policy Changes • Look for Budget Changes and Transitions to Hard Monies • Look for Personnel Shifts and Changes in Job Descriptions • Look for Hard Resource Allocations

Summary • Understand Context • Understand the Program or Project • Be Flexible but Rigorous • Ask a Lot of Questions • Report What You See

Questions and Discussion Contact: boltll@appstate.edu or boltll@comcast.net

Evaluation Realities: What the Theories and Standards Do Not Say