1 / 15

TAH Project Evaluation

TAH Project Evaluation. Great - Yet Realistic - Expectations Jeff Sun, Sun Associates. Our Background. Initiative-wide Qualitative Evaluation District-level initiatives State-level initiatives Strategic Planning STEM, Science, Instructional Technology, Social Studies, New Media

gittel
Download Presentation

TAH Project Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TAH Project Evaluation Great - Yet Realistic - Expectations Jeff Sun, Sun Associates

  2. Our Background • Initiative-wide Qualitative Evaluation • District-level initiatives • State-level initiatives • Strategic Planning • STEM, Science, Instructional Technology, Social Studies, New Media • Evaluation of Teacher Professional Development • 10 TAH projects (7 of these are current)

  3. Qualitative is the Emphasis • Makes the connection between what projects say they want to do and what participants say that projects have done. • This data is invariably qualitative and accounts for a wide range of effects • Quantitative data helps paint the picture in broad outlines... • ...but qualitative data helps you understand what the quantitative data means and to use the evaluation strategically

  4. This Presentation • We want to talk about how to • Create a good evaluation • Use evaluation strategically • Work effectively with evaluators

  5. Four Functions for Evaluation • Evaluation performs four meaningful functions in a project’s life... • As a part of the proposal • To clarifying a project’s work • To provide formative assessment of Progress and Implementation • Summatively, to account for the project’s success (or lack thereof)

  6. First Function - The Proposal • As expressed in the logic map, evaluation helps explain: • Need for the project • The project’s actions in response to those needs • Who the project will serve • What the project expects to produce • All in one tidy package in the proposal text!

  7. Issues Related to Function 1 • We need time to develop the evaluation plan • Ideally, we come in when the basic work is set, but the details are not fully fixed • We need space to fully describe the evaluation process • It’s hard to create this plan in 2 paragraphs or a half page • We need connect with those who will be actually implementing the project if funded • Proposal-writers will help you get funded...but it’s often no coincidence that they’re nowhere to be found once you get the grant.

  8. Second Function - Clarification • Once you’re funded, the real work begins • Use the evaluation plan as an organizer for talking with partners about • what the project will be doing • expectations for partners and participants • This discussion clarifies goals and develops a shared vision for project success • Out of this discussion come the project’s performance indicators...which we organize into rubrics • It would be nice to have all of your indicators before funding, as part of the proposal • But realistically, there’s never time for that • Projects need their own indicators • ED’s indicators are not the same and cannot replace your own indicators

  9. Issues Related to Function 2 • We can’t help if we aren’t at the table • Often, we have to really bug projects to bring us on board early • Evaluation is too often seen as the thing that happens after the work is done...not as the thing that can get work done. • We read the proposal and base the actual evaluation on what is written • This can be a problem when the proposal was written by the now-absent proposal-writer • The clarification discussion - assuming it happens early enough - is where this problem is resolved

  10. Third Function - Formative • Collecting the data and sharing it as the project progresses • Measuring what the project is doing against what it said that it would do • Using indicator rubrics to account for data that is highly qualitative

  11. Issues Related to Function 3 • Our challenge is to identify data collection points that get at the essence of the project and its impact. • This can be one of the most important parts of our work. • Questions continue the process of clarifying the project work through participant feedback on impact • Next most difficult is connecting directly with participants

  12. Fourth Function - Summative • Annual comparison of progress to goals • Sometimes more reporting than “annual” is important • More reporting might be useful in the beginning, less reporting might be necessary as the project matures • Summative data should play a role in the development of new proposals

  13. Issues Related to Function 4 • Qualitative data is valuable for projects, but does not seem to be something that ED is interested in. • This is a problem because projects cannot really produce meaningful data that ED is interested in. • So, do we spend extra time (money) producing two sets of data? • Or do we sacrifice the meaningful data to concentrate on producing what we’re told to create? • This has been the on-going debate among TAH evaluators on a national level

  14. Questions? Comments? • What challenges have you faced when considering project evaluation? • How have you used evaluation in different stages of your project? (from planning to reporting) • jsun@sun-associates.com or 978-251-1600 x 204

More Related