1 / 38

Evaluating and Reporting Program Impacts: Contentious Public Issues

Evaluating and Reporting Program Impacts: Contentious Public Issues. Loretta Singletary Professor & Extension Educator University of Nevada Cooperative Extension. Staci Emm University of Nevada Cooperative Extension

kelleyk
Download Presentation

Evaluating and Reporting Program Impacts: Contentious Public Issues

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating and Reporting Program Impacts: Contentious Public Issues Loretta Singletary Professor & Extension Educator University of Nevada Cooperative Extension

  2. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Common Assumptions • Can’t measure impacts of “process” programs because… • Handing out evaluations is awkward • Participants don’t like to complete forms • I don’t need to measure the impacts because I know when it is successful • I don’t want to report results of little to no impacts or negative impacts

  3. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Common Assumptions • Can’t measure impacts of “process” programs because… • These programs are just too complex • There is too much controversy • Some participants arrive angry, stay angry and leave angry • My role is limited to that of facilitator only • My role is that of educator only • My role is fact-finder and researcher only

  4. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Dilemma • All affected stakeholders do not see collaboration as a better alternative • Stakeholders do not participate equally • Participants have different skills, commitment, motives

  5. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Dilemma • Playing field is uneven • Stakeholders lack sufficient time and receive varied “dosage” • External forces are rushing the process • Funding and time constraints are problems

  6. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluation Required? • LOGIC model is a systematic method for program planning that requires (handout): • Situational Analyses • Priority Setting • Action Plan • Evaluation

  7. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Benefits of Evaluation • Serve to strengthen programs over the short and long term • Evaluations determine strategies and methods that work best, under which conditions and for which purposes

  8. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Benefits of Evaluation • Enable comparisons with traditional decision-making processes to build public understanding of consensus-based processes

  9. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Evaluating Impacts of Programs that Address Contentious Public Issues • WHAT? • WHO? • WHEN? • HOW?

  10. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 What to Evaluate? • Learning objectives = desired changes • Programs that address contentious public issues try to accomplish some basics: • Educate • Communicate • Explore options • Seek mutually satisfying agreements

  11. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 What are Desired Changes? • Contentious issue efficiently addressed • Multiple options developed • Multiple options considered • Climate becomes less contentious • Climate supports public discourse • Conflict is resolved for mutual satisfaction

  12. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Primary Impacts(Gray, 1989) • Two objective criteria to measure impacts of a collaborative effort: • Negotiated agreement is reached • Agreement has been implemented

  13. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impacts • Subjective criteria show impacts of collaboration • Alter individuals’ skills to participate • Alter perceptions of others • Alter perceptions of issue • Alter attitudes • Alter behavior • Develop criteria to measure these changes

  14. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impact Measures • Improved… • Communication skills • Networking skills • Relationship building skills • Increased hope for resolving the issue

  15. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impact Measures • Quality of Interaction • Volunteers learn how to share power • Treat one another fairly and with respect • Operational Details • Meetings follow timeline • Meetings accomplish goals • Participants stay informed (Gray 1989)

  16. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Secondary Impacts… • First order vs. second order effects • Second order effects… • Increased knowledge about issues • Increased awareness of issues • Increased awareness of diverse views • New personal and working relationships • Stakeholders accept and understand scientific analyses • Respond to future conflict civilly (Innes, 1999) • Best when retrospectively measured

  17. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 When to Evaluate? • Midcourse – partway through to assess progress and improvements • End-of-process – at conclusion to assess learning, satisfaction and first order effects • Retrospective – 12 to 18 months after process to identify value and stability of process and second order effects

  18. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Midcourse Evaluation • After a few months of process program to assess “how it is going”… • Are process criteria being met? • Are participants satisfied? • Are representatives communicating with their stakeholder groups? • Do all participants feel well informed? • Do all participants feel equally empowered? • Are participants engaged and interested?

  19. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 End-of-Process Evaluation • Within month or two of final meeting • Experience still “fresh” in everyone’s mind • Objectively assess accomplishments • Participants feel time well spent? • Participants plan to stay involved with implementing agreement? • Participants would try this type of process again?

  20. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Retrospective Evaluation • Of single process takes place at least one year after process completed • Examines immediate and longer term effects of program • Traces over time program effects on participants and their relationships to one another

  21. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Who to Evaluate? • Program objectives determines WHO • Joint fact finding • Policy dialogue • Public-private partnerships • Regulatory negotiation • Advisory Committees • Public Forums • Seeking negotiated agreement

  22. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Who to Evaluate? • Program objectives determines WHO • Process participants (stakeholders—those directly involved) • Agencies not directly involved • Citizens not directly involved • Funding sources • Program speakers, researchers, facilitators

  23. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • Focus Groups (least expensive) • Face-to-face interviews • Postal surveys • Internet surveys

  24. Principles for Writing Questions • Simple wording, yet complete sentences • Avoid vague quantifiers • Avoid specificity (how many) • Equal scalar (positive and negative) • Provide “Don’t Know” category for each question * • State both ends of scale equally (Poor / Excellent) • Eliminate “check all that apply” • Avoid open ended questions • Offer ordered responses (scale) Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate?

  25. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Close-ended questions with ordered responses provide participants with a response scale; Select one answer from a scale or fixed range of choices.

  26. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? All citizens have a responsibility to conserve water on a daily basis.  Strongly disagree  Somewhat disagree  Neither agree nor disagree  Somewhat agree  Strongly agree

  27. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • My ability to listen to others’ views has: •  Improved very little •  Improved little • Stayed about the same • Improved much • Improved very much

  28. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • My understanding of the effect of salinity on the Lahonton Cutthroat Trout fishery has: •  Improved very little •  Improved little • Stayed about the same • Improved much • Improved very much

  29. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Close-ended questions with unordered responses present answers in no particular order and participants must pick the one best response.

  30. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? Whose responsibility is it to see to it that watershed water quality improves? (check only one):  Area farmers  Environmental Protection Agency  My Regional Watershed Planning Group  Local elected officials  Environmental interest groups  Every citizen living in the watershed area

  31. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 How to Evaluate? • An example of a partially closed-ended question with unordered response is: • How do you prefer to receive ABC Program information? (check all that apply) •  Printed material mailed to me •  Small group meetings •  Large public forums •  Newspaper articles •  Other (please describe) ________________________________

  32. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting QuantitativeImpacts • Ranked means using Likert scale • Percentages collapsing highs/low Likert scale • Regression analyses using participant characteristics to find impact patterns • Several other tests for relationships

  33. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting QualitativeImpacts • Frequency count by key words • Verbs • Nouns • Direct objects • Categorize comments by class • Liked most • Liked least

  34. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Impacts • Citizen participants • Participating agencies • Your administrators • Land-grant universities (academic body) • Funding sources

  35. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Reporting Impacts • 4-page fact sheets • In-depth Extension publications • Power point presentations to citizens • Professional conferences (ANREP, NACAA, NACDEP, ESP, Galaxy) • Refereed journal articles (JOE) • News articles • Radio and television interviews

  36. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Bibliography • Bickman, L., Rog, D.J. & Hedrick, T.E. 1998. Applied Research Design: A Practical Approach in L. Bickman and D.J. Rog (eds.) Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications. • Dillman, D. A. 2000. Mail and Internet Surveys: The Tailored Design Method, 2nd edition. New York: John Wiley and Sons. • Dillman, D. A. 1978. Mail and Telephone Surveys: The Total Design Method. New York, NY: John Wiley and Sons. • Gray, B. 1989. Collaborating: Finding Common Ground for Multiparty Problems

  37. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Bibliography • Heron, J. 1996. Co-operative Inquiry: Research into the Human Condition. Thousand Oaks, CA: Sage Publications. • Innes, J. E. 1999. Evaluating Consensus Building. Thousand Oaks, CA: Sage Publications. • Patton, M.Q. 1978. Utilization-Focused Evaluation. Beverly Hills, CA: Sage Publications. • Salant, P. & Dillman, D.A. 1994. How to Conduct Your Own Survey. New York: John Wiley and Sons. • Singletary, L. 2004. Mail Surveys, in L. Singletary (Ed.) Conducting Community Situational Analysis: A Field Guide to Dynamic Extension Programming. Reno, NV: UNCE EB-04-02,available online at www.unce.unr.edu.

  38. Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Staci Emm University of Nevada Cooperative Extension 314 5th StreetP.O. Box 810 Hawthorne, NV 89415(775) 945-3444fax – (775) 945-2259 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary University of Nevada Cooperative Extension 504 S. Main Street P.O. Box 811 Yerington, NV 89447-0811 (775) 463-6541 fax – (775) 463-6545 Loretta Singletary, Ph.D.Professor and Extension Educator University of Nevada Cooperative Extension Phone: (775) 463-6541 FAX: (775) 463-6545 Email: singletaryl@unce.unr.edu

More Related