Program design and evaluation a training series for nonprofits serving boulder county
This presentation is the property of its rightful owner.
Sponsored Links
1 / 50

Program Design and Evaluation A Training Series for Nonprofits Serving Boulder County PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on
  • Presentation posted in: General

Program Design and Evaluation A Training Series for Nonprofits Serving Boulder County. Session 1: Introduction to Evaluation and Creating a Logic Model. Acknowledgements. Boulder County Human Services Strategic Plan: http://buildinglivablecommunities.org/

Download Presentation

Program Design and Evaluation A Training Series for Nonprofits Serving Boulder County

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Program design and evaluation a training series for nonprofits serving boulder county

Program Design and EvaluationA Training Series for Nonprofits Serving Boulder County

Session 1: Introduction to Evaluation and Creating a Logic Model


Acknowledgements

Acknowledgements

  • Boulder County Human Services Strategic Plan: http://buildinglivablecommunities.org/

    The Community Foundation, Foothills United Way, Boulder County, The City of Longmont, The City of Boulder, The City of Lafayette

    OMNI Institute (www.omni.org)


Learning objectives

Learning Objectives

  • Describe evaluation basics, including terms and types of evaluation

  • Depict your planned program work and intended results by constructing a logic model

  • Understand how a logic model can drive selection of research design and data collection methods


Agenda

Agenda

1. Basics of Evaluation

2. Create a Logic Model

3. Overview of an Evaluation Plan

4. Upcoming Trainings and Technical Assistance


Basics of evaluation

Basics of Evaluation

Answer this question:

When I think about

program evaluation I feel….


Program design and evaluation a training series for nonprofits serving boulder county

What is Evaluation?

Why is it

important to

YOU ?


Evaluation is

Evaluation is…

…careful collection of information about a program – or some aspect of a program – in order to make necessary decisions about the program.

(C. McNamara, 1998)


Definition of evaluation

Definition of Evaluation

A systematic collection of information about your strategy

  • Process (activities, characteristics…)

  • Outcomes (change/impact)

    That results in

  • Reduced uncertainty

  • Improved effectiveness

  • Informed decision-making


Benefits of evaluation it s about utility

Benefits of Evaluation: It’s About Utility

  • Reflect on progress – where you are now, where you came from, and where to go…

  • Ensure that program objectives match activities and outcomes

  • Influence policy makers and funders

  • Share what does and does not work

  • Ensure funding sustainability

  • Strengthen accountability


Types of evaluation

Process

Describes the “who,” “what” and “how” of program services. It answers the question, “Is the program being implemented as expected?”

Outcome

Describes the extent of change (e.g. behaviors or knowledge) as a result of the program. It answers the question, “What was the impact of the program on the clients served?”

Types of Evaluation


Process and outcome activity

Process and Outcome Activity

Worksheet: Process, Outcome and Impact

Note: “impact” refers to an intermediate or long-term outcome (e.g. change that occurs more than 12 months following the program and up to 4-6 years out).


Steps taken in evaluation

Steps Taken in Evaluation

1. Create Logic Model

2. Develop an Evaluation Plan

3. Identify an Appropriate Method for your Outcomes

4. Collect & Enter Data

5. Analyze Data

6. Report and Reflect on Data

Evaluation is an ongoing process.


Program design and evaluation a training series for nonprofits serving boulder county

‘Would you tell me, please, which way I ought to go from here?’

‘That depends a good deal on where you want to get to, ’ said the Cat.

‘I don’t much care where’ said Alice.

‘Then it doesn’t matter which way you go,’ said the Cat.- Lewis Carroll


Logic model in its simplest form

How do we get there?

Where are we now?

Where do we want to be?

Logic Model in its Simplest Form


What is a logic model

What is a Logic Model?

  • A simplified picture of a program, initiative, or intervention that is a response to a given situation

  • It shows the logical relationship between the need you will address, the services you will provide, and the changes you expect to see as the result

  • It is at the core of program planning, evaluation, program management and communication


Program design and evaluation a training series for nonprofits serving boulder county

Benefits of a Logic Model

  • Develops understanding

  • Helps monitor progress

  • Serves as an evaluation framework

  • Helps expose assumptions

  • Helps restrain over-promising

  • Promotes communication


Creating a logic model

Creating a Logic Model

Problem:Issue or situation to be addressed

Services:Activities to change the problem

Outcome:Changes or benefits that result

Problem /

Situation

Services

Outcomes


Creating a logic model1

Creating a Logic Model

(Silly!) Example

Problem

Services

Outcomes

Take pain reliever pills as directed

No more headache!

Severe Headache


Program design and evaluation a training series for nonprofits serving boulder county

Simplified Logic Model & Terms

Problem /

Situation

Services

Outcomes

Strategies

Activities

Outputs

Client

Characteristics

And Needs

Outcomes

Measurable Indicators


Definitions

Definitions


Program design and evaluation a training series for nonprofits serving boulder county

Needs Assessment

Choose Program Model (research)

Simplified Logic Model

Client

Characteristics

And Needs

Strategies

Activities

Outputs

Outcomes

Measurable Indicators


Theory the missing link

Overall Theory Base

Links the outcome or issue you want to impact to potential causes

Program Theory Base

Links chosen program to a change in the “causal factors”.

Theory: The Missing Link

Early Academic

Failure

School

Dropout

Research and Evidence

Early Intervention

(preschool model)


Example school drop out research base

Example: School Drop Out (Research Base)

“Causal Factors” or “Predictors” (research-based)

  • Transitions and Mobility

  • Poverty

  • Early Academic Failure

  • Lack of school commitment or “connectedness”

    Evidence-Based or Research-Based Strategies

  • Positive Behavior Support

  • Project ACHIEVE

  • Positive Action

  • Perry Preschool Project


Theory of change discussion

Theory of Change Discussion

Small Groups:

  • What is an example of research that you utilize to inform your program/strategy?

    Large group share outs


Components of the boulder county program logic model

Components of the Boulder County Program Logic Model

  • Program Goal

  • Target Population

  • Program Services

    • Resources dedicated

    • Type of service

    • Extent (or amount) of the service

  • Impact of the program

    • Short-term, intermediate and long-term outcomes


Program goal

Program Goal

A goal is a broad, generalized statement about what is to be accomplished in the long-term.

Goals

  • Focus on a condition or outcome of interest – based on real community need

  • Establish the direction and reasoning of a strategy

  • Avoid defining the problem in terms of a preferred solution

  • Are framed in a way to build consensus


Goals examples

“Good”:

To increase early academic success in Alexander school district

“Needs Work”:

To provide tutoring to at-risk students in Alexander elementary school

Goals: Examples


Target population

Target Population

Answers the question: What types of clients will and won’t

our program be effective with?

Individual

characteristics

Environmental

Variables

Abilities

Systems

Involvement


Program design and evaluation a training series for nonprofits serving boulder county

  • Target Population

Also answers the question: What types and level of need will and

won’t our program be effective with?

Expresses both

LevelandType

Of Need


Example school drop out target population

Example: School Drop Out(Target Population)

Example 1: 3-5 year olds with low SES (Socio-Economic Status) in Alexander School District

Example 2: 3rd graders not reading at grade-level in Alexander School District


Logic model activity goal statement and target population

Logic Model Activity:Goal Statement and Target Population

Individual work

  • Review/refine or develop goal statement

  • Review/refine or identify target population

    Large group share outs


Impact of the program

Impact of the program

Short-term and Intermediate Outcomes

  • Measures the steps taken that will lead to a change in the long-term outcome such as: knowledge, attitudes, and skills acquired or changed as a result of services

    Long-term Outcomes (Impact)

  • Require more time to be achieved and are typically more difficult to connect directly to services due to outside influences that also impact the outcome

Client

Characteristics

And Needs

Strategies

Activities

Outputs

Outcomes

Measurable Indicators


Outcomes and goals definitions

Outcomes and Goals: Definitions

Short-Term Outcomes: immediate effects of the program (changes in knowledge, attitudes, skills or level of functioning).

Intermediate Outcomes: longer term effects of program that are generally achieved one year after the implementation of the program or strategy (changes in knowledge, attitudes, skills or level of functioning).

Long-Term Outcomes or Impact: achieved generally in 4-6 years, and usually show change in behavior, norms, policies, organization and systems.

Goals: state the overall purpose of the program, expressed in terms of changes.


What is a measurable outcome

What Is A Measurable Outcome?

  • Describes the extent of change (how much)

  • Identifies the target population(for who)

  • Specifies the behavior, condition, or knowledge you hope to change (what)

  • Includes a date or time frame by which the change can be expected (by when)


Example school drop out measurable long term outcome

“Good”

25% reduction in early academic failure at Alexander School District within the next 5 years

Reduction in early academic failure at Alexander School District within the next 5 years

“Needs Work”

Encourage school completion

Example: School Drop Out (Measurable Long Term Outcome)


Example school drop out goal and outcomes

Example: School Drop Out(Goal and Outcomes)

  • Short term outcome: All 80 students who receive the Perry Preschool program met at least 90% of the 58 key developmental indicators upon completion.

  • Intermediate term outcome: 80% of students who participated in Perry Preschool program are on-track academically in the first grade; 80% of students who participated demonstrate less antisocial behavior and misconduct in the first grade.

  • Long-term outcomes: 25% reduction in early academic failure in Alexander School District within the next 5 years

  • Goal: Increased early academic success in Alexander School District


About outcomes

About Outcomes

  • There is no right number of outcomes.

  • Strategies more directly affect short term outcomes than longer term outcomes.

  • Don’t discard an outcome because it’s affected by other forces.

  • Outcomes should not go beyond the strategy’s purpose or target population.


Program design and evaluation a training series for nonprofits serving boulder county

Logic Model Activity

  • Develop short, medium or long-term outcome on the logic model.

  • Considerations:

    • Is there consistency (logical link) between the short-term, intermediate and long-term outcomes?

    • Is there consistency between the goal and the long term outcome?


Program design and evaluation a training series for nonprofits serving boulder county

Where We Get Confused

Getting lost in the timing of outcomes – “Is it an outcome or a short-term outcome or is it really a long-term outcome … no, it’s a goal … I think?”

Understanding how our “theory of change” or research-base fit into a model


Services and resources

Services and Resources

  • Type of Service or “Activities”: the processes, tools, events, technology and actions that are part of program implementation (answers the question: what service are we providing?)

  • Extent of Services or “Outputs”: direct products or deliverables of the program, such as number of classes held, number of hours of the program, number and type of participants, etc… (answers the question, how much of this service are we providing?)

  • Resources: human, financial, organizational, & community resources (answers the question, with what resources are we providing this service?)


Programs services

Programs/Services

Link with:

  • Target population: needs & characteristics

  • Program goal(s) and outcomes

    Quality attributes:

  • Appropriate for target population

  • Evidence-based or research-based

  • Sufficient intensity and duration to achieve expected outcome


Program design and evaluation a training series for nonprofits serving boulder county

Examples:Resources, Activities, Outputs

Answers the question: What are we providing, how much,

and with what resources?


Example school drop out services and resources

Example: School Drop Out (Services and Resources)

  • Resources dedicated:

    • Early childhood education teachers, elementary schools, trained coach, evaluation tools.

  • Type of service:

    • 1) curriculum, small class sizes and individual-assessments; 2) quality staff, learning environment and quality assessments; and 3) support for low SES families and parent/guardian engagement.

  • Extent (or amount) of the service:

    • 80 preschool-aged children with low SES; 36 week curriculum; home visit; 3 CORs per child; 4 PQAs; 2 parent-teacher conferences, monthly newsletters and bi-monthly family gatherings.


Review creating a logic model

Review: Creating a Logic Model

  • Involve program stakeholders

  • Identify the goals of the program

  • Specify the target population

  • Identify short and longer-term outcomes

  • Outline the services you provide and the resources dedicated to the program

  • Identify the service amount or outputs


Use of logic models

Use of Logic Models

Once you’ve created a logic model…

  • Use it to plan your evaluation

    • How will you measure your outputs?

    • How will you measure your outcomes?

    • How will you monitor your implementation?

  • Review and update it regularly


Steps taken in evaluation1

Steps Taken in Evaluation

1. Create Logic Model

2. Develop an Evaluation Plan

3. Identify an Appropriate Method for your Outcomes

4. Collect & Enter Data

5. Analyze Data

6. Report and Reflect on Data

Evaluation is an ongoing process.


Learning objectives1

Learning Objectives

  • Describe evaluation basics, including terms and types of evaluation

  • Depict your planned program work and intended results by constructing a logic model

  • Understand how a logic model can drive selection of research designs and data collection


Evaluation

Evaluation

  • Please complete your evaluation of today’s session

  • THANK YOU!


Technical assistance from omni

Technical Assistance from OMNI

  • OMNI trainers will be available on May 29th from 9am to 4pm to meet with individual organizations to answer questions specific to your programs

  • Sign-up for a one hour slot through The Community Foundation. Registration opens this Friday and is open until all slots are filled

  • Email [email protected] and we will respond to specific questions you may have related to the content covered in this training


Future trainings

Future Trainings

  • Selecting Evidence-based Programs, Practices, and Policies

  • Practical Evaluation Design: Developing a Plan (or How to Get Started)

  • Practical Evaluation Design: Collecting your Data

  • Program Refinement: Using your Results


  • Login