Tailoring Evaluations

1. Tailoring Evaluations Karen (B.K.) Adler

2. What Aspects of the Evaluation Plan Must be Tailored? • The questions that need to be answered • The methods and procedure the evaluation will use to answer those questions • The nature of the evaluator-stakeholder realtionship

3. What features of the Situation Should be Taken Into Account? • Purposes of the evaluation • Program structure and circumstances • Resources available

4. The Purposes of the Evaluation • Program improvement • Program effects • Decisions about funding • Response to political pressures

5. Purposes of the Evaluation • Program improvement • Accountability • Knowledge generation

6. Program Improvement – Formative Evaluation • Audiences • Program planners • Administrators • Oversight boards • Funders – interested in program effectiveness • Information Desired • Need for program • Program design • Implementation • Impact • Efficiency

7. Accountability – Summative Evaluation • Effective use of resources • Beneficial contribution to society • Audiences • Decision Makers • Funding boards • Governing board • Legislative committee

8. Knowledge Generation • Test theory • Try new approaches • Audiences • Sponsors of research • Interested scholars • Policymakers

9. Hidden Agendas • Good public relations • Rqationale for a decision that has already been made • Delay tactic to appease critics • Watch out for political and public relations motivations!

10. Program Structure and Circumstances • Stage of program development • Status quo called into question? • Administrative and Political Context of the Program • When major stakeholders don’t agree… • Guba and Lincoln – interpretative dialog among stakeholders

11. Conceptual and Organizational Structure of the Program • Program Theory • Sometimes difficult for stakeholders to articulate a program that is well established and routine • Large complex programs – Difficult! • New math curriculum • Discrete or concrete – Easy! • Serve meals to homeless

12. Resources Available • Personnel • Materials, equipment • Facilities to support data collection, analysis and reporting • Source of funding? Different aspects of evaluation funded differently?

13. Other Critical Resources • Support from program management • Access to information and program records • Ability to read and understand the records • Cooperation from program personnel • TIME! Better to answer a few questions well than a lot poorly

14. The Nature of the Evaluator-Stakeholder Relationship • Policy and decision makers • Program sponsors • Evaluation sponsors • Target participants • Program managers • Program staff • Program competitors • Contextual stakeholders • Evaluation and research community

15. Some Good Advice • Identify stakeholders • Involve stakeholders early • Involve stakeholders continuously • Involve stakeholders actively • Establish a structure for the sharing of information.

16. Types of Relationships • Independent Evaluation • Participatory or Collaborative Evaluation • Empowerment Evaluation

17. Communication and Dissemination of Findings. • Varied formats • Short reports, summaries, presentations • Tailor content to the audience • Use clear language • Graphs, charts, vivid, concrete illustrations • Present contextual information • Include both positive and negative findings

18. Evaluation Questions and Methods • Give structure to the evaluation • Lead to appropriate and thoughtful planning • Basis for essential discussions about how the answers will be used

19. 5 types of Questions • Needs assessment • Assesment of program theory • Assessment of program process • How’s it working – most frequent type of eval. • Impact assessment • Efficiency assessment

20. Conclusion • Every evaluation must e tailored to a specific set of circumstances • Purposes of the evaluation • Questions and methods • Resources available • Evaluator-stakeholder relationship