1 / 51

Bronte Koop, Principal Consultant, Strategy and Evaluation

Telling the Difference between Snake Oil and Good Oil; Evaluation Capacity Building in the public sector. Bronte Koop, Principal Consultant, Strategy and Evaluation John Collins, Manager, Program Evaluation. National Partnerships Literacy and Numeracy Teacher Quality Low SES communities.

italia
Download Presentation

Bronte Koop, Principal Consultant, Strategy and Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Telling the Difference between Snake Oil and Good Oil; Evaluation Capacity Building in the public sector Bronte Koop, Principal Consultant, Strategy and Evaluation John Collins, Manager, Program Evaluation

  2. National Partnerships Literacy and Numeracy Teacher Quality Low SES communities COAG Reform Agenda: Human Capital Development

  3. Not getting what you expected from an evaluation

  4. What do we mean by Snake Oil • Staff as passive recipients • Evaluation questions not addressed • Use lots of words • Recommendations to continue the program and evaluate it again • Lack of evidence based recommendations for improvement • No discussion of data or design limitations • No discussion of alternative explanations for program effect • Evaluators reluctant to address client concerns about clarity or focus

  5. Useful information to guide decision making

  6. So what’s the Good Oil • Staff critical participants • Provides answers to specific questions • Balances qualitative with quantitative data • Reports are just sufficient in length • Conclusions follow from the evidence • Has logical recommendations • Data or design limitations are discussed • Alternative explanations for program effect are discussed

  7. How do we get there? teamwork commitment learning proactivity

  8. Attitudes and values What are we trying to change? Knowledge and skills Organisational behaviour

  9. Overcoming your fears is the first step

  10. Principles for evidence based program management • Demand evidence • Examine the logic • Treat the program as an unfinished prototype • Embrace an attitude of wisdom • Jeffrey Pfeffer and Robert Sutton on evidence based management

  11. CREATINGCHANGE Where should be put our efforts? • Executive leadership • Consulting Services • Resources • Professional development • Policy and operations • Networks • Strategic Oversight

  12. Locate near the top Recruit the knowledge, skills and motivation Build credibility Provide an ‘evaluation perspective’ Consulting Services

  13. Resources If you build it, they will come

  14. What have others done?

  15. Accessible Resources

  16. Practical Resources

  17. Practical Resources Provide a starting point

  18. Project Management

  19. Evaluation & planning

  20. Professional Development • Time to stretch your thinking • Induction • Customized training to needs

  21. Policy and Operations Think outside the square Procurement: Expression of Interest Two part contracts: plan + do Executive summaries Interviews Hard questions

  22. Network, network, network

  23. Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

  24. Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

  25. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . .

  26. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate.

  27. Strategic Plan 2012-2016 We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate. DRAFT

  28. Represents a major cultural change.

  29. Represents a major cultural change • From an ‘accountability’ model to one centred on the corporate goal.

  30. Represents a major cultural change • From an ‘accountability’ model to one centred on the corporate goal. A tough call

  31. How?

  32. A marketing approach

  33. A marketing approach Product life-cycle

  34. A marketing approach Product life-cycle

  35. A marketing approach Product life-cycle Customer

  36. A marketing approach Product life-cycle Customer Needs and wants

  37. A marketing approach Product life-cycle Customer Needs and wants

  38. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

  39. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

  40. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

  41. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase

  42. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity

  43. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity

  44. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity Competition

  45. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity Competition

  46. A marketing approach Product life-cycle Customer Needs and wants The grudge purchase Brands and brand equity Competition 4 Ps – Profit Product Place Promotion

  47. Priorities? To continue development & refinement of our ‘system’ To address changing thefocus of our evaluation culture To move from ‘good oil’ to ‘best value’ oil

  48. Disclaimer The views and interpretationsexpressed in this presentation are those of the authors anddo not necessarily reflect those of the Department of Education and Children's Services or the Government of South Australia

More Related