Based on work by Lydia B. Blalock, Ph.D., Youth Development Specialist New Brunswick, New Jersey. Evaluation Based Program Development. Karen Barale, Pierce County Renee Overath, Kitsap County Christy Price, Grant County. From Assessment to Impact: Day 1. Introduction. Why are you here?
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Based on work by Lydia B. Blalock, Ph.D., Youth Development Specialist New Brunswick, New Jersey Evaluation Based Program Development
Karen Barale, Pierce County Renee Overath, Kitsap County Christy Price, Grant County From Assessment to Impact: Day 1
Introduction • Why are you here? • When I hear Program Evaluation I think . . . • Education Philosophy • Experiential: I can’t teach anything, but I can provide opportunities to learn! • There’s no such thing as a stupid question!
How We Got . . . Here • Various backgrounds • Program Evaluation Philosophy • We can’t develop or perform an evaluation if the program wasn’t built upon a strong foundation! • “Train the trainer” Evaluation-Based Program Development
Process Objectives If I: • Develop and presentseminar on EBPD • Develop and distribute appropriate handouts • Engage participants in group activities • Provide participants an opportunityto practice • Encouragequestions THEN
ProgramDevelopment Understand how to clearly define issue program will address. Use program theory to plan activities. Write clear outcome and process objectives. Use EBPD Model to plan programs. Outcome Objectives Participants will:
Program Evaluation Develop skills to evaluate programs. Select appropriate evaluation methods and designs. Select criteria to evaluate objectives. Use EBPD Model to plan evaluation. Outcome Objectives Participants will:
Scholarship of Engagement Understand scholarship opportunities in EBPD process. Identify several avenues to share work. Use EBPD to plan scholarly activities. Cultural Aspects of EBPD Understand influence of culture on EBPD (personal & audience). Understand importance of learning culture of target audience/s. Select evaluation methods appropriate for diverse audiences. Outcome Objectives Participants will: Commit to putting new knowledge and skills into practice!
BUT FIRST: Is it a Program or a program • Stakeholder leadership • Faculty leadership • Multi-disciplinary • Sustainable • Academic department • Student component • Research, teaching, outreach
Group Assignments • Form groups of 4 • Sit with new group • Select bag • Develop program based upon contents of bag • Develop 10-15 minute (tops) program presentation
Program Development Project: Requirements • Define problem • What method/s did you use • Identify stakeholders/collaborators • Develop outcome objectives • Target audience • Resources needs • Short, medium, long term objectives
Program Development Project: Requirements • Plan program activities (process objectives) • Develop evaluation plan • Success indicators • Research design and data collection methods • Identify scholarship opportunities
Step 1. Define the Problem • Describe Current Situation • How does the world look today? • Why a Problem • Clearly articulate why a problem • Ask why until get to root of problem • Symptom vs. Cause
Step 1. Define the Problem • Identify Stakeholders • Population: Who is affected? How many? • Stakeholder Leadership: interest in solving problem? • Potential Collaborators: Already working on problem?
Step 1. Define the Problem • How: Evidence? Learn more? References & Facts • Learn all you can about the issue/problem – • “prove” the problem exists • Identify potential solutions • “Needs” Assessment • Need: difference between actual and target “state” • A.K.A.: Asset, Gap, etc.
Small Group Activity:Needs Assessment Needs Assessments • Purpose of “needs” assessment is to clearly define or clarify issue. • What kinds of information might you need? • Who or what can provide the information? • What methods could you use to gather information? • Where can you gather or find the information?
Needs Assessments • How will we gather the information? • Surveys • Interviews • Focus groups • Card sort • Archival/secondary data • Criteria for selecting methods • Consider characteristics of target and respondent groups • Choose combo of methods to gain different types of info • When • Develop target dates and timelines
Using Results • Set Priorities • Factors that influence priorities • Difference between actual and target states • Degree of difficulty in addressing needs • Cost of implementing solutions • Political and other factors • Derive priorities from several data source • Priorities without resources
Using Results • Select Solution Strategies to Meet Needs • Consider alternative solutions for each need • Evaluate alternatives • Select solutions to implement • Scholarship Opportunities?
Step 2. Develop Outcome Objectives Ï • Who do you want to change? • What do you want to accomplish? • Do you have resources to accomplish changes?
Step 2. Develop Outcome Objectives Ï • Describe Ideal Situation • How would world look w/o problem (perfect world)? • Define Target Audience/s • Who/what can we change (youth, families, policies, etc.)? • What do we know about culture/s of audience?
Step 2. Develop Outcome Objectives Ï • Identify Outcome Objectives • What changes can I influence in my target audience/s? • Identify Resources • What do we need? • What do we have?
Writing Objectives • Specific • Will increase knowledge vs. Will know 3 sources of Vitamin C • Time delineated • At the end of the program, in 6 weeks, etc. • Measurable • 75%, all, etc.
Writing Objectives • Example • Participants will increase knowledge of nutrition. • 75% of program participants will know 3 sources of Vitamin C 6 weeks after the program.
Step 3. Plan Program Use “Program Theory” to develop process objectives!
Program Theory: The Key to Unlocking Change
WORKING LUNCH • Define the problem • Outcome objectives • Process objectives (program activities)
I dunno I think I knew why–once They make me do it Department chair Funding organization My mom I don’t Why Evaluate?
Demonstrate success Understand failure Provide feedback for program improvement Process Outcome Nip problems in the bud Why Evaluate?
Why We Don’t Evaluate • Too hard • Not qualified • Costs too much • Takes too much time • Don’t think about it “in time” • No one ever reads evaluation reports • Everyone “knows” it’s a good program
Establish criteria What are criteria (benchmarks, indicators) for judging your item? Construct standards How well should your item perform on benchmarks (standards)? Develop criteria rating system Small Group Activity: Setting Criteria
AFTER developing criteria: measure performance and compare with standards. How well does each type of item measure up against criteria? Small Group Activity:
Process Indicators • How will you know you’ve successfully accomplished your process objectives?
Step 5. Implement Program and Conduct Evaluation • You planned your work. • Now is the time to work the plan. • If all goes well – great! • If problems arise – check plan, revise as needed
Cultural Aspects of EBPD • What is culture? • “Behaviors and beliefs characteristic of a specific social, ethnic, or age group.” • Product of our culture • Culture influences (unconsciously) what we think of people from other cultures. • Recognize cultural biases towards participants and strive to eliminate them.
Cultural Aspects of EBPD • Neutralize cultural biases and prevent embarrassing (or worse) EBPD design flaws is: • Know your audience (Step 2 – Develop Outcome Objects) • Be aware of how differences between cultures may impact EBPD • Develop cross-cultural efficacy!
Step 6. Dissemination of Results - Extension Scholarship • Tenure track faculty engage in appropriate scholarly activities • Scholarship • Development and implementation of creative concepts . . . • . . . broadly communicated and validated by appropriate peers