National Strawberry Sustainability Initiative

1. National Strawberry Sustainability Initiative Reporting Project Outcomes

2. Part 1 Thinking about Your Project Reporting

3. Results Based Program: Outcomes & Impacts Your project was selected because it was determined to have the potential to make significant impact on sustainable strawberry production. • NSSI – a results based program; the grant selection, reporting, and management The Goal: To increase the probability that funded programs will improve the skills, practices and technology of the strawberry growers and industry that participate. • Key components (Logic Model based): • Outputs • Outcomes • Impacts • Evaluation

4. Beginning with the “end” in mind Projects should Begin with the End in Mind • Determine first what the outcome is that the participants will achieve – or you want them to achieve. • Design the program (outputs) to lead the participants to those results/outcomes. • “How will you get there?” • Evaluation – how will you measure the change?

5. Logic Model: Beginning With the End in Mind INPUTS (Resources that will strengthen participant achievement of results) OUTCOMES (Identify Measurable Technology/ Practice Results) OUTPUTS (Lead project participants to the desired results) Logic Model Implementing Forward INPUTS OUTCOMES OUTPUTS From: Western Risk Management Education Center, http://westrme.wsu.edu/award-management/

6. Logic Model as a Tool to Construct Realistic Outcomes First: Identify the situation(s) that your project will help to address (pest management, input reduction, food safety etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer/Industry Demand). Then: • Outcomes – Identify the changes in knowledge, actions, or conditions/system you would like program participants to realize • Outputs– Identify activities (workshops, on-farm trials, demonstrations, etc.) that will support and lead to participant outcomes • Inputs – Resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc.

7. An Outcome Defined…. Progressive Change in: • Knowledge* • Actions, Attitudes* • Condition, System, Environment *Useful Tool: Search Bloom’s Taxonomy of Action Verbs Or See: http://www.clemson.edu/assessment/assessmentpractices/referencematerials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf

8. Outcomes Defined…. Changes in knowledge… • When project participants Learn or become aware of new fundamental or applied knowledge which could include: • Practices and methods • Technology • Improved skills such as decision making, evaluating

9. Outcomes Defined…. Change in Actions… • When project participants Act upon what they have learned such as: • Implementing new practices, methods or technology • Changing or modifying a practice • Adoption of improved skills • Direct application of knowledge gained

10. Outcomes Defined…. Change in Conditions, System, Environment… • A condition is changed because of industry/growers wide-spread actions • Increased number of strawberry producers in a state/region leading to increased production and availability. • Increased local/regional strawberry production or season extension to increase availability to consumers. • Adoption of practices that leads to a reduction in greenhouse gas emissions. Etc.

11. Outcomes Will Depend Upon: • Knowledge of your target audience • Why industry/producers will choose to participate • Action and learning processes (project steps) that help industry/producers achieve long term outcomes • Program resources including the team, collaborators and expert presenters sufficient to lead participants to the desired outcomes.

12. Producer Results or Outcomes an example of progressive steps… • Understand or have improved understanding of the issue, the practice/technology/methods or the need to take action. • Decide to take an action. - Producer makes a commitment to write a business or marketing plan, set up a record keeping system, or initiate other steps that will lead to adoption • Develop –customize a system or plan • Implement or take an action that improves farm or business i.e. – implementing a food safety plan, using a new technology, etc.

13. In Summary… Outcomes • Measure of success for the participant • Ability to change/incorporate/implement/adopt practices, technology or skill, etc. as a result of participation in your project. • Achievable and measureable within a reasonable timeframe • Sequential • Identifies changes in condition or practice you seek for the participant • Specifies the degree of change required for participant success • Stated in a measurable way knowing methods you will use to verify changes in practices, condition, etc.

14. Part 2 NSSI Reporting

15. strawberry.uark.edu NSSI Reporting Questions

16. NSSI Reporting: Outputs “List your outputs in relation to the objectives of the proposal. Include activities such as presentations (professional, academic, outreach, stakeholders, consumers, etc.), publications (professional, academic, outreach, etc.), workshops, websites, blogs, etc. Include target audience and number of people.” What activities did you conduct to move your participants towards project outcomes?

17. NSSI Reporting: Outcomes “List the outcomes of your project, i.e. what change in knowledge, attitude or actions did your outputs accomplish in the short to intermediate time frame. Include who was reached, how many were reached and how outcomes were measured.” • What did your participants/target audience do with the project outputs? • Increase knowledge? Initiate change in thought processes? Change behaviors/practices? • How did you measure this?

18. NSSI Reporting: Impact “List the actual or anticipated impact of your project, i.e. the big-picture effect of your project on the industry/growers/region etc. (changes in condition, environment, society) over the longer-term. Include how impact were or will be measured.” True impacts are harder to measure and may not be evident until late in the project or after the project has been completed.

19. Other NSSI Reporting Questions • Summary of your project • Objectives of your project • Media contacts • Internet media • Problems delays or changes in plans • Next accomplishments • Attachments/Uploads

20. Underperformance • Performance will be evaluated based upon your timely and thorough reporting • Underperforming projects risk termination. • Performance metrics include; • Timely activities as proposed • Accomplishments towards your project goal • Timely and appropriate use of funds • Termination • Unsatisfactory project performance will receive notice • Notice will include corrective actions and reporting • Project have one reporting period to correct • Reminder: Project Deadline is June 30, 2014

21. NSSI Reporting and Submitted Financials • Project Progress Reports should be in alignment with your Project Financial Reports. • The progress, outputs, and outcomes of your project will be compared to what funding has been used and its purpose.

22. Evaluation

23. Evaluation How will you measure the effect of your outputs to ensure they are creating suitable/appropriate participant change or any change at all? • Protocols • Methodologies • Data collection

24. Measuring Outcomes • Begin with the end in mind • Begin with determining: • The results that will be accomplished • How they will be evaluated • Design evaluation when developing project • Key tip • For each proposed outcome you should be able to write an evaluation question • Do this when you develop your funding proposal • Validates that proposed results are actual results • Helps design the evaluation from the beginning

25. NIFA Metrics for Measuring Outcomes Test your methods • Will your evaluation tool(s) clearly demonstrate progress participants are making towards achieving the outcomes? • Changes in Knowledge? • Changes in Attitude? • Changes in Behavior/Approach? • Changes in Condition or the System?

26. How can you evaluate your outcomes? • Identify measurable expected outcomes. • Define criteria for success. • Develop evaluation tools to measure results. • End-of-training evaluations • Post-training evaluations • Evaluation distribution.

27. Defining Criteria for Success Consider • What will you actually be measuring? • What is the standard for success? • How many participants need to achieve ‘success’ in order to be successful? • When should participants achieve results?

28. Evaluation Mechanics Suitable Question Formats • Multiple Choice – Yes/No • Rating scales • Open ended • Simple for respondent to understand • Easy to analyze for project teams with less evaluation experience Variety of Methods: • Pre/Post Tests • Audience Response Systems (interactive polling Clickers or apps) • Follow-up Evaluations (3 Months – Year) • Mail or Internet Survey • Telephone Interview • Face-to-face interviews (in-person survey) • Dot survey • Informal Interview

29. Evaluation: Pre and Post Test Pre-test Purpose • Benchmark info on knowledge levels • Information on participant background, past experience, attitudes and needs for program managers and instructors Post-test Purpose • Measures • Knowledge Gain • Attitude Change • Actions participants expected to take as a result of participating in workshop/demonstration/project.

30. Combining Pre/Post into Post • “Please rate your knowledge PRIOR to attending today’s workshop” • “Please rate your knowledge AFTER attending this workshop” • “After attending today’s workshop, answer the following questions: • Please rate to what extent you have the knowledge, tools and resources needed to … • I better understand…” • Likert Scale or Similar Method

31. Follow-Up Evaluation Purpose: • To see what actions participants have taken as a result of participating in your project and how these actions have led to benefits for their farm/business.

32. Evaluation Best Practices Measuring results/outcomes successfully: • Choose methods that balance best practices with available resources • Combination of direct and indirect measures • Direct measures (assignments, action plans etc.) yield stronger evidence of change in knowledge and beneficial practices, but are more time intensive and may take more expertise to evaluate. • Indirect measures can give instant feedback but may not provide real evidence of knowledge gained. • Qualitative measures (like phone interviews or focus groups) that may be more resource intensive could be accomplished through a representative sample.

33. Evaluation Best Practices Measuring results/outcomes successfully: • Direct Interactions with Producers can increase the chance that participants will implement beneficial practices/technology etc. • Good follow-up can lead to collegial relationships that may bring results beyond the initial results/outcomes that were initially projected (Unexpected results). • Good program design allows for learning and application to take place in linked stages within a timeframe that encourages the implementation/adoption of longer term results/outcomes. When writing the evaluation questions, think about: • How the information gathered will be used. • The wayyou ask a question will influence how participants respond. • When you ask a question will influence how participants respond.

34. Results/Outcome Resources • The Rensselaerville Institute: http://whatworks.org/displaycommon.cfm?an=1&subarticlenbr=266 “Outcome Funding: A New Approach to Targeted Grant making” Based on the premise that grant makers are really investors in human gain, Outcome Funding addresses the key questions that the funder as investor should ask: What am I buying? What is the probability I will get it? Outcome Funding begins with an abrupt and timely challenge to the standard wisdom of the Request for Proposal process. The authors make the point that very little of the traditional proposal content addresses results, favoring explanations of process instead. This book is not only persuasive on the need for change; it is clear and practical on how to achieve it.  • Developing a Basic Logic Model For Your Program: http://www.publichealth.arizona.edu/chwtoolkit/pdfs/logicmod/chapter2.pdf I have become convinced that it makes a considerable difference if you do the outcomes before planning the activities. I definitely advocate doing the outcomes first! I find that people come up with much more effective activities when they do. Use the motto, “plan backward, implement forward.” Beverly Anderson Parsons, WKKF Cluster Evaluator http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html • Ties to the Logic Model. http://extension.oregonstate.edu/sites/default/files/for-employees/employee-resources/soars-modules/logic_model/index.htm\ • Measuring Impacts for Sustainable Agriculture Programming http://agsci.oregonstate.edu/sites/default/files/understanding_program_development_and_evaluation.pdf

35. Citations • Information in these slides was borrowed and adapted from Western Risk Management Education Center • http://westrme.wsu.edu/award-management/ • How Will Your Verify? – Measuring Producer Results • Extension Risk Management Education • http://www.extensionrme.org/ • Oregon State University, Writing Powerful Impacts • http://agsci.oregonstate.edu/research/writingimpacts

36. National Strawberry Sustainability Initiative Reporting Project Outcomes sberries@uark.edu