1 / 46

Outcomes: Libraries Change Lives — Oh yeah? Prove it.

This presentation by the Public Libraries Association in Phoenix, AZ discusses outcome-based evaluation in libraries and how it can prove the positive impact of libraries on individuals and communities. It covers the definition of outcomes, the evaluation process, and how to use the results.

jamesshernandez
Download Presentation

Outcomes: Libraries Change Lives — Oh yeah? Prove it.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Outcomes: Libraries Change Lives — Oh yeah? Prove it. Public Libraries Association Phoenix, AZ 2002 The Institute of Museum and Library Services

  2. Institute of Museum and Library Services The Institute of Museum and Library Services (IMLS) is a federal agency that fosters leadership, innovation, and a lifetime of learning through grants to museums and libraries. Please see IMLS’s Web site at http://imls.gov for additional information about IMLS and its grant programs.

  3. We will • Talk about outcome-based evaluation (OBE) • Talk about benefits • Define terms • Describe process • Discuss how to use results

  4. What are outcomes? Outcomes are achievements or changes in • SkillInformation literacy Basket weaving • KnowledgeA state’s population Symptoms of diabetes • BehaviorCompletes homework Reads to his kids

  5. What are outcomes? Outcomes are achievements or changes in • AttitudeLibraries are good I support recycling • StatusHigh school graduate Certified librarian • Life conditionHomeless Healthy

  6. Example Wally goes to a reading program and • Learns childhood reading is important • Wants to read to his son • Uses a literacy program • Advances 2 literacy levels • Gets his GED (What kinds of outcomes are each of these?)

  7. Outcomes Where do they come from? • Government Performance and Results Act (GPRA) 1993 • New accountability, need to evaluate • Funding trends – IMLS – State government, LSTA – Foundations (Gates, Pew)

  8. Evaluate? Let me count the Ways! • Internal • External • Participant • Process • Inputs, Outputs (counts) • Outcomes • Formative • Summative • Cost/Benefit • ROI...

  9. How does OBE fit libraries? • Inputs — How much we use • Outputs — How much we do • Performance quality — How well we do • Economic value — What we’re worth • Outcomes — What good we do

  10. How does OBE fit libraries? • OBE changes focus, from activities to benefits • OBE needs to be part of the program design • OBE shows to what extent a program met its own goals

  11. Fears & Realities • Will use too much time – Consider scope of the evaluation • Visitor privacy may be compromised – Ask for volunteers – People like to tell you what they think

  12. Fears & Realities • Takes money from other priorities – Funds are available – Results help leverage other funds • Library impact cannot be measured – Show short-term, immediate impact

  13. More Realities • Libraries collect similar information to improve services and user satisfaction • Evaluation can increase participation, improve services, and leverage funds

  14. Why measure outcomes? • Know if program met purpose • Improve programs • Guide management • Communicate program impact • Satisfy funder’s needto know

  15. What is a program? • Series of services & activities that lead to a goal • Has a definite beginning & end • Meant to change attitude, behavior, knowledge, skill, status, or condition

  16. How to develop a program • Identify a need • Need can be based on • Your experiences • Program partner experiences • Formal or informal research

  17. How to develop a program Look at assumptions & verify needs • Students need structure after school • Students need homework help • Kids like computers • Our librarycan provide safety, structure, computer skills, and help with homework skills

  18. How to develop a program Identify solution A structured after-school program to provide kids with computer skills and homework help Identify desired results • Kids will learn basic computer skills • Kids will be information literate • Kids will get better grades

  19. How to develop a program Look at stakeholders Include individuals, agencies, funding sources, competition, community groups and national and state affiliations. They influence: • Desired outcomes • How results are communicated • Type and nature of services • Who program serves

  20. How to develop a program Look at audience Who is served by a program depends on several factors: • Stakeholders • Assumed need • Mission and resources

  21. Does what For whom For what outcome Example CompuCOOL Program Purpose Provides a M-F after-school computer homework center Kids 8 to 12 in Springfield Kids have basic computer skills, are information literate, and get better grades

  22. Program Elements Services Activities Inputs Outputs Resources What a program uses Tasks What a program does Products Quantity of work or products

  23. Outcome-based program Services Activities Outcomes Outputs Inputs Changes in knowledge, skills, abilities, attitudes, condition or life status

  24. Measuring outcomes • OBE is not formal research • OBE shows contribution, not attribution • OBE shows what results program achieved

  25. Measuring outcomes • Building a logic model • Logic model is the evaluation plan • Shows how all elements fit together • Helps assure desired results

  26. Outcomes Logic Model PROGRAM

  27. Outcomes Identify program outcomes • Immediate • Intermediate • Long term

  28. Outcomes Hints: • A program may have many outcomes • Pick a few important outcomes to measure • One significant outcome may be enough

  29. Outcomes CompuCOOL Outcome 1 Participants have basic computer skills Outcome 2 Participants are “information literate” Outcome 3 Participants complete homework

  30. Indicators • Observable, measurable, clearly identifiable • Unambiguous • Several may apply to each outcome

  31. Indicators CompuCOOL Outcome 1: Participants have basic computer skills Indicator: # and % of participants who can word- process one complete assignment without error

  32. Indicators Does using a word processor indicate basic computer skills? • A word processor is basic tool • Word processing requires specific skills • If kids can produce a short assignment, they probably have one basic skill set

  33. Indicator Apply to Data source Data interval Target # and % of kids who can word process 1 complete assignment without error All kids who finished “basic” training session Completed assignment After “basic” training session 75% Logic Model CompuCOOL Outcome 1: Kids have basic computer skills

  34. Reports What should reports say? • We wanted to do what • We did what • So what

  35. Reports • Summarize participant characteristics • Summarize inputs, activities/services, outputs, and outcomes • Respond to influencers’ need for information • Compare data from program start or previous period • Interpret results and make recommendations

  36. Inputs • What did we use? • How much did we spend? • How much did we consume? Activities & Services • What did we do? Outputs • How many units did we deliver? • To whom? (audience characteristics) Outcomes • What did we achieve for our target audience? Report Elements

  37. Who will do the work? Options: Hire a consultant Benefits • Result may be seen as unbiased • Professionals have most expertise • Process may be more efficient • Offers outside perspective

  38. Who will do the work? Options: Hire staff evaluator Benefits • May reduce cost • Greater understanding of your program • Greater commitment to the process

  39. Who will do the work? Options:Train existing staff(ideal) Benefits • Integrate evaluation into routine management activities • Staff know programs and audience • Skills transfer to all programs

  40. What do you get? • Small investment: Numbers, audience characteristics, and customer satisfaction • Low to moderate investment: Immediate changes in knowledge, behaviors, and attitudes

  41. What do you get? • Moderate to high investment:Attribute short-term changes in audience skills, knowledge, behaviors, and attitudes to program • High investment: Short- and long-term impact, attribution of impact to program, variables influencing impact

  42. Your Action Plan • Discuss benefits to your library • Get more information • Consider how to apply OBE • Develop a plan

  43. We have • Talked about uses of OBE • Talked about benefits • Defined terms • Described process • Discussed how results are used

  44. Starting Places: • Perspectives on Outcome-based Evaluation for Libraries and Museums(2001), IMLShttp://www.imls.gov/pubs/pdf/pubobe.pdf • IMLS bibliography http://www.imls.gov/grants/current/crnt_obe.htm#res

  45. Starting Places: • Action Plan for Outcomes Assessment in Your Library, Peter Hernon and Robert E. Dugan (2002), ALA Editions A comprehensive plan specifically for librarians; provides data collection tools for measuring learning outcomes that link outcomes to user satisfaction • W.K. Kellogg Foundation Evaluation Handbook(1998) http://www.wkkf.org/pubs/Pub770.pdf Thorough introduction for organizations new to OBE

  46. For more information Karen Motylewski Institute of Museum and Library Services 1100 Pennsylvania Avenue, NW Washington, DC 20506 202-606-5551 kmotylewski@imls.gov http://www.imls.gov

More Related