Assessment evaluation and value current trends
This presentation is the property of its rightful owner.
Sponsored Links
1 / 74

Assessment, Evaluation, and Value: Current Trends. PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on
  • Presentation posted in: General

Assessment, Evaluation, and Value: Current Trends. Rosalind F. Dudden, MLS, DM/AHIP, FMLA Library Services Director Tucker Medical Library National Jewish Medical and Research Center Denver, Colorado 80206. Outline. Background Trends in Assessment Trends in Evaluation

Download Presentation

Assessment, Evaluation, and Value: Current Trends.

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Assessment evaluation and value current trends

Assessment, Evaluation, and Value: Current Trends.

Rosalind F. Dudden, MLS, DM/AHIP, FMLA

Library Services Director

Tucker Medical Library

National Jewish Medical and Research Center

Denver, Colorado 80206


Outline

Outline

  • Background

  • Trends in Assessment

  • Trends in Evaluation

  • Outcomes Measurements

  • Reporting Outcomes

  • Value concepts

C E N D I Principles Meeting


Assessment evaluation and value current trends

  • “When I read about the way in which library funds are being cut and cut, I can only think that American society has found one more way to destroy itself.”

    • Isaac Asimov (Russian born American science-fiction Writer and Biochemist. 1920-1992)

C E N D I Principles Meeting


Response to library cuts

Response to Library Cuts

  • National Library of Medicine Publications Grant, now called NLM Grants for Scholarly Works in Biomedicine and Health

  • (#5-G13LM008520) Oct 2004 to Sept 2006

  • Purpose: a book-length manuscript (or other scholarly work) of value to U.S. health professionals.

  • A good score - hope that other people thought the book would be of use to librarians in small library settings.

C E N D I Principles Meeting


Response to library cuts1

Response to Library Cuts

Table of Contents

Part I: Evaluating Library

Quality and Performance

1. Why Evaluate?

2. The Effective Library

3. Library Measures

New York: Neal Sculman, July 2007.

C E N D I Principles Meeting


Response to library cuts2

Response to Library Cuts

  • Part II:Working with Evaluation Methods

    4. Method 1: Needs Assessment

    Workbook for Needs Assessment

    5. Method 2: Quality Improvement

    Workbook for FOCUS-PDCA

    6. Method 3: Benchmarking

    Workbook for Performance Benchmarking

    Workbook for Process Benchmarking

    7. Method 4: Library Performance Standards

    Workbook for Library Performance Standards

    Workbook for Accreditation Standards

C E N D I Principles Meeting


Response to library cuts3

Response to Library Cuts

8. Method 5: Outcomes Measurement

Workbook for Describing Published Studies of Outcomes Measurement

Workbook for Cost Outcomes

Workbook for the Logic Model

9. Other Systems for Quality Improvement and Evaluation

Part III: Tools for Doing Evaluations

10. Data Collection and Analysis Methods

11. Skills for Communicating in Evaluation Projects

12. Tools for Improvement and Evaluation

Appendix A: Glossary of Terms

Appendix B: Recommended Reading

C E N D I Principles Meeting


Straight path of evaluation

Straight Path of Evaluation

C E N D I Principles Meeting


But soon

But Soon…

Cover - The New Republic - 2/19/2007

C E N D I Principles Meeting


Background

Background

  • It started with Lancaster

  • Lancaster FW. The Measurement and Evaluation of Library Services. 1977. 1991 (with Baker SL).

  • Lancaster FW. If You Want to Evaluate Your Library. 1988, 1993

C E N D I Principles Meeting


Background process

Background - Process

  • Libraries would do a process evaluation

    • No one would notice except those working in the part of the library being evaluated.

  • Processes were improved, customer needs met, the head librarian managed.

  • While not rich, libraries in general were supported.

  • Libraries were considered “essentially good.”

  • Why evaluate something that is going to exist anyway because of its basic value to society or the corporation?

C E N D I Principles Meeting


Background comparative

Background - Comparative

  • To assure or perhaps to prove quality, library associations renewed their interest in writing library standards and guidelines.

  • At the same time ARL and AAHLS started collecting comparative data on library operations and still do today.

  • MLA followed recently by collecting data for non-academic health sciences libraries in their Benchmarking Network surveys.

C E N D I Principles Meeting


Background technology

Background - Technology

  • 80s - Technologically, libraries started feeling a paradigm shift from the printed word to other media.

  • 90s on - The impact of the Internet on library operations is unprecedented.

C E N D I Principles Meeting


Background integration

Background - Integration

  • The integration of all types of information, as described as early as 1982 in the Matheson report, is now being felt in many settings where libraries operate.

  • Information is being connected on a scale and with a speed often felt to be overwhelming.

    • Electronic health records

    • Industry-wide portals

    • Federated search engines

C E N D I Principles Meeting


Background1

Background

  • Virginia Holtz, in her 1986 MLA Janet Doe lecture, “Measures of Excellence: the Search for the Gold Standard,” when the Internet was just a transmission device, stated what is still true today:

    “Typography of information is primitive. … Further study will continue to reveal its full structure and the relationships among the parts.”

C E N D I Principles Meeting


Background2

Background

What happened to just buying and cataloging a book?

C E N D I Principles Meeting


Shifting paradigms

1900

Engineering

linear development

vertical integration

2000

Biology

diffusion

webs

volatility

adaptability

Shifting Paradigms

C E N D I Principles Meeting


Linear arrangement of types of measures

Linear Arrangement of Types of Measures

Dudden, RF. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-To-Do-It Manual. New York: Neal Sculman, 2007.

C E N D I Principles Meeting


Linear arrangement of types of measures1

Linear Arrangement of Types of Measures

Dudden, RF. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-To-Do-It Manual. New York: Neal Sculman, 2007.

C E N D I Principles Meeting


User behavior

User Behavior

  • Multidimensional nature of the typical graduate student.

  • A person who is struggling to complete his/her dissertation and uses information resources in a multiplicity of ways.

    • Martha Kyrillidou, Director, Statistics and Service Quality Programs, ARL

C E N D I Principles Meeting


User behavior1

User Behavior

  • directly borrow library resources

  • talk to faculty members who have used the library themselves

  • borrow a book from his or her advisor

  • find another interesting resource while visiting a fellow-students' home

  • communicate via e-mail with other experts who refer him or her to additional resources and citations

C E N D I Principles Meeting


User behavior2

User Behavior

  • browse the Internet and locate blogs and wikis in his or her specific field of interest

  • send reference inquiries to ask-a-service operations

  • talk with a fellow student whose spouse works in the library and discuss with him or her the articles he or she was able to locate in the library

  • - - with the help of a librarian

C E N D I Principles Meeting


Inputs to outputs to outcomes

Inputs to Outputs to Outcomes

“This inextricable and complex web of relations provides a richer context for the usefulness of inputs, outputs, quality issues, and the impact libraries are making.”

  • Martha Kyrillidou. From input and output measures to quality and outcome measures, or, from the user in the life of the library to the library in the life of the user. J Acad Libr. 2002;28(1/2):42-6.

C E N D I Principles Meeting


User in the life of the library

“User in the life of the library..

  • “The usual linear model (Figure 2) depicting a linear, sequential, rational, and controlled set of relations

  • is indeed limiting our understanding of a model (Figure 3) that is more cyclical, haphazard, and uncontrolled.”

C E N D I Principles Meeting


The library in the life of the user

“the library in the life of the user…

  • “This becomes even more complex when one tries to introduce the notion of motion (Figure 4) depicting a more dynamic and flexible model,

  • moving users and information resources

  • into a spiral swirl up and down

  • into the depths of knowledge, exploration, and experience.”

C E N D I Principles Meeting


What do we measure

What do we measure?

  • Needs

  • Inputs

  • Quality Processes

  • Outputs

  • Quality Service

  • Outcomes

  • Impacts

C E N D I Principles Meeting


Assessment evaluations

Assessment - Evaluations

  • Definitions:

    • “Assessment is the gathering of meaningful or purposeful data that will provide information that informs, improves, or confirms.”

    • “Evaluation is assigning merit, value or worth to the findings.”

      • Lee-Thomas G, Robson J. The questions of academic library assessment. Indiana Libr. 2004;23(1):6-10.

C E N D I Principles Meeting


Measure needs

Measure Needs

  • Needs assessment

  • Customer focus

    • Perley CM, Gentry CA, Fleming AS, Sen KM. Conducting a user-centered information needs assessment: the Via Christi Libraries' experience. J Med Libr Assoc. Apr 2007;95(2):173-81, e54-5.

C E N D I Principles Meeting


Needs assessment results

Needs Assessment Results

  • Clinicians emphasized the need for

    • “Just in time” information accessible at the point of care

    • Information services customized to their professional information needs, preferences, and patterns of use

    • Organization-specific information needs identified

  • Library nonusers emphasized the need to market library services and resources.

C E N D I Principles Meeting


Measure inputs

Measure Inputs

  • What is the size of our staff, budget, space, or collection?

  • How do we compare to others in benchmarking studies?

  • Often criticized, but comparisons are sometimes it is the only thing stakeholders understand.

  • The comparisons answer questions people ask because they do not know what else to ask.

C E N D I Principles Meeting


Measure inputs1

Measure Inputs

  • A casual conversation:

  • “How many e-journals do you have?”

  • Answer 1: “350 (or whatever) AND did you know that the process of investigation by our scientists has been fast-tracked by their access to this information at their desktop?”

  • Answer 2: “1500 (or whatever) AND did you know that the use of these resources by our high school students at home has increased their ability to have more resources cited in their papers?”

C E N D I Principles Meeting


Measure inputs2

Measure Inputs

  • You still need those traditional input counts.

    • Good management

    • Budgeting and resource allocation

    • Comparative Benchmarking

C E N D I Principles Meeting


Measure quality processes

Measure Quality Processes

  • TQM (Total Quality Management) - now part of any management culture.

  • Quality improvement techniques:

    • PDCA - Plan-Do-Check-Act

    • Gemba Visits

    • Kaizen

    • Lean, Five Ss

    • Learning Organization

    • Sigma Six

    • Balanced Scorecard

C E N D I Principles Meeting


Measure quality processes1

Measure Quality Processes

  • David Orenstein

  • A quality culture “considers service issues, people, and challenges as simultaneous and interconnected concerns.”

  • Eleven ways to move your library to adopt a quality culture -

    • Orenstein DI. Developing quality managers and quality management: the challenge to leadership in library organizations. Libr Adm Manag. 1999;13(1):44-51.

C E N D I Principles Meeting


Quality culture

Quality Culture

  • Build a shared vision for the library.

  • Put the needs of the customers before the politics of the organization.

  • Build cooperation among all levels of employees.

  • Communicate.

  • Emphasize teamwork.

C E N D I Principles Meeting


Quality culture1

Quality Culture

  • Build trust.

  • Redesign processes and attitudes.

  • Train for quality.

  • Develop leadership skills.

  • Manage by fact. – (EBLIP4.unc.edu)

  • Motivate staff by making work enjoyable.

C E N D I Principles Meeting


Measure outputs

Measure Outputs

  • Answers the question - What do you do?

    • Use in communications to stakeholders

  • Identify trends.

  • Can indicate a re-assignment of staff.

  • A trend up or down may indicate the necessity of further evaluation of a program or service.

C E N D I Principles Meeting


By the numbers

By the Numbers

  • 8,428,033Items circulated

  • 417Full-time equivalent staff members

  • 3,918,658 Visitors to the Central Library and 22 branch locations

  • 31,378,982Transactions conducted online

  • 99,332Hours donated by docents and volunteers

  • 939,393Reference questions answered system-wide

  • 22,446Kids / teens who registered for Summer of Reading

    (2005 Annual Report)

C E N D I Principles Meeting


Measure service quality

Measure Service Quality

  • LibQual+™ - from ARL - http://www.libqual.org/

  • A suite of services that libraries use to solicit, track, understand, and act upon users’ opinions of service quality.

  • The program’s centerpiece is a rigorously tested Web-based survey - bundled with training that helps libraries assess and improve library services, change organizational culture, and market the library.

C E N D I Principles Meeting


Measure service quality1

Measure Service Quality

  • LibQual+™ seeks to measure three dimensions of library service quality:

    • the library as place (utilitarian space, symbol, refuge);

    • information control (scope, timeliness, convenience, ease of navigation, modern equipment); and

    • affect of service (empathy, responsiveness, assurance, reliability).

C E N D I Principles Meeting


Measure service quality2

Measure Service Quality

  • Non-ARL libraries

  • Simple satisfaction surveys

  • Rate service as to

    • Satisfaction 1 2 3 4 5

    • Importance 1 2 3 4 5

  • Used Web-based tools such as SurveyMonkey or Zoomerang

C E N D I Principles Meeting


Measure outcomes

Measure Outcomes

  • Started in 1993 with the Government Performance and Results Act (GPRA)

  • The GPRA required that most federal agencies develop objective, quantifiable and measurable goals and report how well those goals were achieved.

  • The influence of the law spread to local governments and philanthropic organizations.

  • If a program says it will change the lives of a certain population group, how does it prove it actually did?

C E N D I Principles Meeting


Measure outcomes1

Measure Outcomes

  • United Way of America:

    • Measuring Program Outcomes: A Practical Approach, 1996.

    • The Outcome Measurement Resource Network

  • W.K. Kellogg Foundation:

    • W.K. Kellogg Foundation Evaluation Handbook, 1998;

    • Logic Model Development Guide. 2004.

    • The Evaluation Toolkit

C E N D I Principles Meeting


Measure outcomes2

Measure Outcomes

  • A fiscally conservative trend

  • “Prove” that the dollars spent helping people really do

  • Trickle down effect to all types of libraries, as well as museums, colleges, and universities

C E N D I Principles Meeting


Outcomes public libraries

Outcomes - Public Libraries

The Institute for Museum and Library Services (IMLS) describes an outcome as

- a benefit to people through achievements or changes in skill, knowledge, attitude, behavior, condition, or life status.

C E N D I Principles Meeting


Outcomes public libraries1

Outcomes - Public Libraries

  • How libraries and librarians help: A Guide to Identifying User-Centered Outcomes, by Durrance and Fisher

  • “Political pressures are very strong for public libraries to prove their worth in competition for tax dollars.”

    • Information School, University of WashingtonSchool of Information, University of Michigan

C E N D I Principles Meeting


Outcomes public libraries2

Outcomes - Public Libraries

  • Outcomes Tool Kit 2.0 provides “guidance for going beyond reporting outputs and will help you to discover outcomes, or indicators of impact, of your programs that can be shared with others.”

C E N D I Principles Meeting


Outcomes public libraries3

Outcomes - Public Libraries

  • Shaping Outcomes

  • An outcome is a change in a target audience’s skills, attitudes, knowledge, behaviors, status, or life condition brought about by experiencing a program.

    • Indiana University Purdue University Indianapolis

    • http://www.shapingoutcomes.org/course/index.htm

C E N D I Principles Meeting


Outcomes ice cream socials

Outcomes - Ice Cream Socials

  • Example: A local Denver dairy is awarding cash grants or ice cream socials to school libraries.

  • As part of the application, the library needs to:

    • Describe who will benefit from the grant.

    • What measure will you use to determine the success of the project?

C E N D I Principles Meeting


Outcomes academic libraries

Outcomes - Academic Libraries

  • Challenges

    • Is the academic performance of students improved through their contact with the library?

    • By using the library, do students improve their chances of having a successful career?

    • Are undergraduates who used the library more likely to succeed in graduate school?

    • Are students who use the library more likely to lead fuller and more satisfying lives?

      • ACRL Task Force on Academic Library Outcomes Assessment

C E N D I Principles Meeting


Outcomes health libraries

Outcomes - Health Libraries

  • Impact federal health library outreach programs

  • Measuring the Difference : Guide to Planning and Evaluating Health Information Outreach. 2000; three supplemental publications, 2006.

  • National Network of Libraries of Medicine Pacific Northwest Region

C E N D I Principles Meeting


Mla ccml vital pathways

MLA-CCML - Vital Pathways

  • The MLA Vital Pathways Task Force

  • To help hospital librarians communicate with other leaders in their institutions about the true value of librarians and library services.

  • Myths and Truths About Library Services

    • Developed by the Colorado Council of Medical Librarians (CCML) Advocacy Committee and MLA, 2006.

    • http://www.mlanet.org/resources/vital/index.html

C E N D I Principles Meeting


Presenting value

Presenting Value

  • Present the facts or truths

  • Present the sources

  • Present your story

C E N D I Principles Meeting


Myth quality of patient care is unaffected by library services

Myth: Quality of Patient Care is Unaffected by Library Services

TRUTH:

  • Research has shown that libraries improve patient care

    • Weightman and Williamson state:

    • “Research studies suggest professionally led library services have an impact on health outcomes for patients and may lead to time savings for health-care professionals.”

C E N D I Principles Meeting


Source

Source:

  • Weightman AL, Williamson J. The value and impact of information provided through library services for patient care: a systematic review. Health Infor Libr J 2005;22(1):4-25.

C E N D I Principles Meeting


Tucker library story

Tucker Library Story:

  • Physicians use the Tucker Library to make patient care decisions:

    • Quote from a recent survey:

    • “The library is critical to my role as a clinical radiologist. Twice within the past month, I have used it to search for a specific clinical diagnosis, resulting in a change of management for the patient, and education for clinician colleagues.”

C E N D I Principles Meeting


Reporting the outcomes

Reporting the Outcomes

  • From published sources

  • or your own survey --

  • Weightman and Williamson report

  • “Evidence of the impact of professional library services on health outcomes for patients and time savings for health-care staff is available and can be used to demonstrate the impact of library services to users, managers and funding bodies.”

C E N D I Principles Meeting


Impact of this outcome

Impact of this Outcome

  • “The results suggest clear evidence of an impact of library services on patient outcomes from both traditional and clinical librarian services.

  • The higher quality traditional library studies suggest effects of impacts of between

    • 37 and 97% on general patient care,

    • 10–31% on diagnosis,

    • 20–51% on choice of tests,

    • 27–45% on choice of therapy and

    • 10–19% on reduced length of stay”

C E N D I Principles Meeting


Reporting the outcomes1

Reporting the Outcomes

  • The authors state:

  • “A practical reliable survey methodology could also be used, at a local level, to provide evidence of the impact of the library service on a range of outcomes.”

  • Development of such a tool would be useful to all libraries.

C E N D I Principles Meeting


Outcomes marshall

Corporate study:

Ability to proceed

Make a decision

Create new opportunity

Save time

Save money

Government study:

Meet a deadline

Deal with an emergency

Improve a policy, procedure or plan

Lessen conflict

Save time and resources

Outcomes - Marshall

C E N D I Principles Meeting


Outcomes and impacts

Outcomes and Impacts

  • Jose-Marie Griffiths and Donald Ward King

  • Special Libraries: Increasing the Information Edge. Washington, D.C., Special Libraries Association, 1993.

C E N D I Principles Meeting


Culture of assessment

Culture of Assessment

  • "A Culture of Assessment can be achieved by

    • creating systems and structures that are based on

    • continuous assessment and evaluation

    • in an organizational culture that is

    • customer focused and

    • uses assessment systematically.”

      • Amos Lakos and Shelley Phipps. Creating a culture of assessment: a catalyst for organizational change. Portal. 2004;4(3):345-61

C E N D I Principles Meeting


Culture of assessment1

Culture of Assessment

  • “Librarians have to create customer-responsive environments that are designed to enhance service quality and maintain superior standards of service.

  • This can only be achieved by creating systems and structures that are based on continuous assessment and evaluation.”

    • Phipps SE. Beyond measuring service quality: learning from the voices of the customers, the staff, the processes, and the organization. Libr Trends. 2001;49(4):635-61.

C E N D I Principles Meeting


Value multiple meanings

Value - Multiple Meanings

  • An ideal accepted by an individual or group. [Librarians value information]

  • The quality (positive or negative) that makes something valuable. [Knowledge of another language had value to her career.]

  • To regard highly or hold dear. [ I value his judgment]

  • To place a value on or judge the worth of something or to fix or determine the value of or assign a value to. [The value of the jewelry is…]

C E N D I Principles Meeting


Assessment evaluation and value current trends

From http://www.visualthesaurus.com/

C E N D I Principles Meeting


Value and benefit outcome and impact

Value and Benefit,Outcome and Impact

  • Value

    • The importance or preciousness of something, the perception of actual or potential benefit.

  • Benefit

    • The helpful or useful effect that something has.

C E N D I Principles Meeting


Value and benefit outcome and impact1

Value and Benefit,Outcome and Impact

  • Outcome

    • the consequence, visible or practical result or effect of an event or activity.

  • Impact

    • the effect or influence of one person, thing, or action, on another.

C E N D I Principles Meeting


Value and benefit outcome and impact2

Value and Benefit,Outcome and Impact

  • Library Outcomes

    • the eventual result of using library services, the influence the use had, and its significance to the user.

      • Poll, Roswitha. Measuring impact and outcome of libraries. Perform Measur Metrics. 2003;4(1):2-12.

C E N D I Principles Meeting


Value

Value

  • Marshall JG. Determining our worth, communicating our value. Libr J. 2000;125(19):28-30.

  • Marshall JG. Valuing Ourselves and Our Work in the Information Age. University of North Carolina. <www.sla.org/Presentations/sldc/joanne_LAB2002pp.ppt>

C E N D I Principles Meeting


Value concepts marshall

Value concepts - Marshall

  • Components of a Librarian’s value proposition

    • the information user

    • the processes used to provide info access

    • research and discovery

    • technology as a tool

    • the human and financial resources required

C E N D I Principles Meeting


Practicing librarians

Practicing librarians

  • Practicing librarians in “small” settings need to be generalists.

  • They hire, fire and evaluate personnel but they are not human resources experts.

  • They develop budgets and financial plans but they are not accountants.

  • They develop mission statements and strategic plans but they are not planning experts.

C E N D I Principles Meeting


Practicing librarians1

Practicing librarians

  • They can learn to do evaluation and assessment without being experts in that field.

  • They are experts in information storage and retrieval, with all the details that involves.

  • The experts in the field of library and information science evaluation and assessment will help all librarians report outcomes and impacts to their stakeholders.

C E N D I Principles Meeting


Current trends

Current Trends

  • Understand the goals of the organization

  • Understand the user - be customer focused - assess needs

  • Establish a culture of assessment

  • Use a variety of measures - old and new

  • Use outcomes to tell your story

C E N D I Principles Meeting


Assessment evaluation and value current trends

  • Full Bibliography and PowerPoint:

  • http://info.nationaljewish.org/libraryinfo/

C E N D I Principles Meeting


  • Login