Why assess? What is assessment? What do we mean by actionable data?

1. Why assess? What is assessment? What do we mean by actionable data? Martha Kyrillidou Planning for Results: Making Data Work for You The Cato Institute, Washington, DC February 20, 2008

2. What’s bringing us together …

3. Overview 1. Initiate (Why do we assess?) “Why assess? What is assessment? What do we mean by actionable data?” - Martha Kyrillidou (1&2) 2. Plan (What do we assess?) “Survey Design” – Neal Kaske (2&3) “Performance Measures” – Rick Garcia (2&3) 3. Design (How do we assess?) 4. Implement (Design in action) “Analysis and Interpretation” - Ray Lyons (5) LibQUAL+®, MINES™, and Other Tools; Packaging Results to Communicate to Stakeholders Gary Roebuck and Kristina Justh (2,3,4,&5) 5. Analyze and Interpret (What do the results mean?) 6. Reflect and Initiate (What do we do with the results?)

4. Clear and present danger • Bangor University considers removing librarians posted by Blake on Thursday January 27, 2005 @07:30AM -753 hits   Ms Information writes "News from the University of Wales Bangor in the UK. senior management no longer feel that subject librarians / academic liaison librarians are needed in the modern academic library. They have made restructuring proposals which include removing all but one of the subject librarians and a tier of the library management, including the Head of Bibliographic Services. The university management thinks that technology has 'deskilled' literature searching. As far as I know, this proposal is unprecedented in the United Kingdom. In essence, there will remain 4 professional librarians serving a 'research-led' university of 8,000 plus FTEs and with 8 library sites. These will be the university librarian, cataloguing librarian, acquisitions librarian and Law librarian. • Has anything like this happened anywhere that you know of? If so, what have been the effects?

5. Assessment Librarian

6. Library of the Future Will Also Need . . . . . . To have it’s own data collection and management personnel, individuals who constantly collect, analyze and prepare reports on data regarding what services are being used, which portions of the collection are getting the highest usage, what materials are being lent through interlibrary loan, and who patrons are. Documenting the libraries contributions to quality teaching, student outcomes, research productivity will become critical.

7. Technical Skills needed • Quantitative Analysis: SPSS • Qualitative Analysis: Atlas.ti • Data collection: Custom Web interface, Survey Monkey, FormSpring • Data storage: SQL Server, Excel • Reporting: Crystal Reports, SPSS, SQL Reporting Services, Excel, Word • Web delivery: Dundas Charts • Web development: ASP.NET, C#, ColdFusion

8. Communication Skills needed • Communicating organizational vision and service ethic • Leadership in capturing the future and making it happen • Ability to take initiative and willingness to take risks and learn from mistakes • Troubleshooting and being an independent thinker • Ability to work collaboratively and move forward team and organizational goals

9. . . . chart a course for every endeavor that we take the people's money for, see how well we are progressing, tell the public how we are doing, stop the things that don't work, and never stop improving the things that we think are worth investing in. • President William J. Clinton, on signing the Government Performance and Results Act of 1993

10. What’s driving the ‘assessment’ agenda • “Times are changing” • Exploding growth in use and applications of technology • Increased customer expectations for services, including quality and responsiveness • Shrinking budgets • Justification for spending $$$ on resources, programs, and services • Increasing competition for resources • Cost savings and/or reallocation • Fighting to remain relevant

11. The Internet Goes to College Early data from ethnographic interviews • “I use Google because I heard it searches for more things” (than other sources). • “I believe I can find anything on the Internet. There hasn’t been anything I haven’t been able to find.” • “Because I’m lazy.” • Books have “so much information that no one can go through it all.” • I use “the Internet first because it is more convenient.” • I go to the library “because that’s what teachers like.” • “Google has gotten me through college.” Source: Steve Jones, The Internet Goes to College, ARL Talk

12. Libraries Remain a Credible Resource in 21st Century 98% agree with statement, “My … library contains information from credible and known sources.” Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.

13. OCLC Reports • Sharing, Privacy and Trust in Our Networked World • College Student’s Perceptions: Libraries and Information Resources • Perceptions of Libraries and Information Resources • OCLC: http://www.oclc.org/reports/sharing/default.htm

14. ‘Universal’ library brand • Amazing stability of library user’s desired expectations across • Types of users • Languages • Countries • Some differences across disciplines • 297,158 LibQUAL+® survey respondents

15. Towards what end The emerging library user sees the need for a different kind of library – a library that serves as a source of information ready to be accessed any time, from anywhere, using modern technologies. Can we effectively translate the comfort and assurance of human interactions into a virtual reality that is trustworthy, reliable, and ever present? And will learning be quite as effective or deep in this virtual world of never-ending interaction with the collective digital-knowledge base? • Thompson, Kyrillidou, Cook “Library Users’ Service Desires” Library Quarterly (2008)

16. Library Assessment and its Global Dimensions • Markets and people exposed to economic and social frameworks unheard of before • Competing internationally • Library users exposed to global forces • Libraries facing similar challenges • Libraries as the Internet • Libraries as Google • Libraries as Collaborative Spaces

17. Defining Success in a Digital Environment • Crafting new measures of success. • Moving from measuring inputs to outputs and outcomes • Understanding impact of library roles and services. • Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing organizational goals. • Reallocating and managing capabilities to focus on new definitions of success.

18. Thinking Strategically About Libraries’ Futures • What is the central work of the library and how can we do more, differently, and at less cost? • What important set of services does the library provide that others can’t? What new roles are needed? • What advantages does the library possess? • What will be the most needed by our community of users in the next decade? How is user behavior changing? • What should our libraries aspire to be ten years from now? What are the implications of technology driven change? • What are the essential factors responsible for the success of the library?

19. Library Assessment Library assessment provides a structured process to learn about our communities, their work and the libraries connection to what they do The information acquired through library assessment is used in an iterative manner to improve library programs and services and make our libraries responsive to the needs of our communities. Academic libraries do not exist in a vacuum but are part of a larger institution. Assessment within the institution may take place in individual areas as well as at the broad institutional level.

20. Assessment is… “…a structured process to learn about communities and evaluate how well the library supports them.” - Steve Hiller, Director of Assessment & Planning University of Washington Libraries

21. Free speech wall, Charlottesville, Sept 2006

22. Evaluation – foci by Peter Brophy Inputs Processes Outputs Outcomes Impacts We have 2 million records As a result of our service, humankind has beaten malaria We have the technology to transform data into information-rich products 26 students have read this paper We have delivered one million documents Increasing difficulty ………… Increasing usefulness …………

23. IFLA: Measuring Quality • Resources, infrastructure: What services does the library offer? • Use: How are the services accepted? • Efficiency: Are the services offered cost-effectively? • Potentials and Development: Are there sufficient potentials for future development? (an adaptation of the Balanced Score Card)

24. Assessment at ARL • A gateway to assessment tools: StatsQUAL®: • ARL Statistics -- E-Metrics • LibQUAL+® -- Service Quality and Satisfaction • DigiQUAL® • MINES for Libraries® • Library Assessment Conferences – libraryassessment.org • Library Assessment Blog – libraryassessment.info • SPEC Kits • Service Quality Evaluation Academy • ESP Assessment (Making Library Assessment Work) • Effective, Sustainable, Practical

25. Qualitative ProfileDeveloping New Metrics (per Yvonna Lincoln) • Uniqueness of collections • Defining the value of consortia • Administrative and budgetary efficiencies • Student outcomes/student learning/graduate success • Contributions to faculty/stakeholders productivity • Social frameworks/intellectual networks • Generating new knowledge • Creating the collective good with reusable assets

26. How do you describe a Library? • Breadth and quality of collections and services • Sustained institutional commitment to the library • Distinctive resources in a variety of media • Services to the scholarly community • Preservation of research resources • Contributions of staff to the profession • Effective and innovative use of technology • Engagement of the library in academic planning Association of Research Libraries ‘Principles of Membership’

27. Group exercise • Share important aspects of your • Services • Collections • Collaborative Relations

28. Tell the story … • Facilities: number and type (library buildings, classrooms, studios, computer labs, cafes, offices outside of the library, etc.) • University centers, services and responsibilities (managed by the library or housed in the library): slide libraries, media centers, teaching and learning centers, education centers, writing centers, distance ed or online course development/instructional design, iTunesU, laptop support, technology help desk, escholarship centers, university press or other publishing entity, university archives, university records management, etc. • Digital collections: institutional/digital repositories, images, articles, simulations, data, learning objects, etc. • Grants and development: numbers, types, friends organizations, advisory boards/councils, etc. • Partnerships: nature and number, on campus and external • Consortial memberships and roles • Outreach: web hits, computer use hours, learning hours/contact/impacts (in classrooms, Info Lit, etc.), visits, external teaching, volunteering, programs offerings, sponsored lectures, etc. • Leadership: on campus and external committees, boards, initiatives, projects, etc. Publications/scholarship: numbers, types (articles, books, presentations, poster sessions, performances, etc.) • Email from ARL Library Director – February 13, 2008

29. Measuring Impact • Concerns Based Adoption Model (CBAM) • Stages of concern • How engaged are subjects in the service /innovation? • How much interest do they show? • Levels of use • How much use do they make? • Avoid dichotomy of user / non-user • Levels of Impact • Draw together concern and use • Assess effects on behaviour • Peter Brophy (South Africa, August 2007)

30. Measuring Impact • Surface learning • Skim, memorise, regurgitate for tests • Little long-term impact • Deep learning • Involved, questioning, interactive, integration with own knowledge • Significant long-term impact • A habit of critical thinking • Peter Brophy (South Africa, August 2007):

31. “If you think you're too small to have an impact, try going to bed with a mosquito.” • Anita Roddick

32. Actionable Data • What is the desired outcome of assessment? • An organization that strives to be customer-centered and actson information about environmental factors and especially on user input. • Raynna Bowlby and Martha Kyrillidou

33. Align with mission, vision and goals of parent organization • Are there prominent themes/directions – research support, instruction, physical facilities; priority user groups; expansion and growth or conservation and reallocation of resources; emphasis on current technologies and methods or more traditional approaches • Are there strategically favored programs/initiatives and/or individual champions with key agendas? • Are there stated goals for the organization and the library?

34. Address users’ top priorities • Are you undertaking initiatives that users indicate are their highest wants and are you doing enough? • What budget/resources are being re/allocated to these initiatives? • Are enough budget/resources being directed to these initiatives? • Have the departments/staff that most affect these initiatives been informed? • Do these departments/staff understand the data (and comments/specifics) and have goals and improvement plans in place? • Are there improvement targets; how is change being measured?

35. Identify resources that can be reallocated • Is the work that you are performing addressing areas of strong dissatisfaction? • Are you doing too much of what you’ve always done, rather than the work that results in user wants & needs? • What budget/resources are being allocated to legacy work that is not a user priority and is it too much? • Have the departments/staff that are currently doing this work been informed of changing user wants and needs? • Do these departments/staff understand the data (and comments/specifics) and can they redirect their time and resources? • Are there improvement targets; is change being measured?

36. Continuous improvement and assessment • How will you incorporate the results of your analysis and findings into your Library’s strategic and annual plans? • How can you develop some of your more resource-intensive improvements over several years? • How can you insure that your assessment results are always available when and where decisions are being made? • How and when will you be assessing the user improvements implemented after your user survey?

37. The importance of accurate data in decision making

38. Your library … • Organization and use of information is focused on disciplines and increasingly interdisciplinary centers • Subject-oriented staff provide information services required and integrate them in the research process • Developing and maintaining relationships with key stakeholders in the disciplines/departments • Acquainting specialized stakeholders with the broad range of information and information services available outside their specialty • Assuring adequate performance of network and technology based information services • Assuring adequate performance of the centralized basic support functions

39. Exercise • What are the key indicators of this library's work? • How do you monitor each indicator? How do you record the data for each indicator? With what frequency? • To whom and how do you communicate information about the library's performance?

41. Making Library Assessment Work • ARL project approved in 2004 and funded by participating libraries • Site visits by Steve Hiller (U of Washington) and Jim Self (U of Virginia) • Presentation • Interviews and meetings • Report to the Library • 24 libraries in U.S. and Canada visited in 2005-06 • Effective, Sustainable and Practical Library Assessment in 2007 • Open to all libraries • 6 libraries participated in 2007

42. What We Found • Strong interest in using assessment to improve customer service and demonstrate value of library • Many libraries uncertain on how to establish, maintain, and sustain effective assessment • Effectiveness of assessment program not dependent on library size or budget • Each library has a unique culture and mission. No “one size fits all” approach works. • Strong customer-focus and leadership support were keys to developing an effective and sustainable assessment

43. What are the lessons learned? • Understanding changes in users approach to information resources. • Service quality improvement is a key factor. • Understanding the impact of e-resources on library services - TRL. • Learning how to compete or collaborate with Google. • Upfront investment in design and development. • Making the assessment service affordable, practical, & effective. • Assessment needs to be satisfying and fun.

44. Multiple methods • Transactional surveys* • Mystery shopping • New, declining, and lost-customer surveys • Focus group interviews • Customer advisory panels • Service reviews • Customer complaint, comment, and inquiry capture • Total market surveys* • Employee field reporting • Employee surveys • Service operating data capture *A SERVQUAL-type instrument is most suitable for these methods Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.

45. Assessment can… • Support decision making • Establish accountability • Strengthen organizational mission, goals, and objectives • Realign library goals and mission statements with the needs of users • Identify trends in user behaviors and technology • Measure learning outcomes and impact • Improve programs and services

46. Use Assessment Data for Good!

48. What’s in a word? What makes a qualitylibrary? “Quality much like beauty is in the eye of the beholder”

49. What’s in a “Library”? A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. --Justice Oliver Wendell Holmes

50. Interpreting Data