Assessment and Evaluation

1. Assessment and Evaluation Accountability in Academic Libraries

2. Focus on Higher Education • Federal government has intensified focus on higher education and its performance: • Started with Spellings Commission • Continues with Obama Adminstration: • DOE criticizes North Central Association of Schools and Colleges • Guide to Accreditation Recognition • Increased focus on “quality” • Time to graduation • Cost

3. Topics • Accountability (institutional effectiveness): state government, accrediting organizations • Perspectives: library, institutional, customer, stakeholders • Types of measures: input, output, customer, outcome (impact) • Outcomes: student learning, research, student • Levels: course, program, college/school, institution • Assessment plan: Which outcomes/measures to focus on

4. Accountability and Institutional Effectiveness • People are asking if the benefits of education equals or exceeds its cost • Federal and state government, the private sector, and education consumers (students and parents) want education institutions held more accountable for the funds, time and other resources allocated and expended in the education process, especially when gaining an undergraduate degree that can be quite expensive

5. Accountability and Institutional Effectiveness • Attention to accountability has resulted in increased attention being given to measures of institutional effectiveness • Traditional measures employed were inputs and outputs. Most recently, outcomes have increased in importance as colleges and universities must demonstrate their effectiveness through efficiency and quality

6. Accountability and Institutional Effectiveness • As a result, there is growing interest in answering such questions such: • What should students learn? • How well are they learning it? • What measures and procedures does the institution utilize to determine that it is effective? • To what extent does the institution offer evidence that demonstrates its effectiveness to the public? • What does the institution plan to do with this evidence to improve outcomes?

7. Accountability and Institutional Effectiveness • Results-oriented questions such as these have resulted in applying outcomes assessment processes in an effort to measure institutional effectiveness. Institutional effectiveness is concerned, in part, with measuring: • accountability (e.g., institutional fiscal efficiency) and • educational quality and improvement (e.g., student learning)

8. Accountability versus Assessment • Simply: • Assessment= when an educational institution or one of its components assesses its own performance • essentially an internal regulatory process, designed to assure institutional conformity to specified norms • Accountability= when others assess performance • accountability is a set of initiatives others take to monitor the results of the institution's actions, and to penalize or reward the institution based on the outcomes

9. Accountability versus Assessment • Assessment for educational quality • address numerous self-evaluative questions related to how well the institution accomplishes its education mission and goals

10. Outcomes versus Learning Outcomes • Student outcomes and student learning outcomes are not the same • Student outcomes refer to aggregate statistics on groups of students (e.g., graduation rates, retention rates, transfer rates, and employment rates for a graduating class) • Such outcomes are institutional outcomes and are used to compare institutional performance • They do not measure changes in students themselves due to their college experience • these outcomes are outputs and reflect what the institution has accomplished; they do not reflect what (or how much) students learned

11. Outcomes versus Learning Outcomes • Student learning outcomes encompass assorted attributes and abilities, both cognitive and affective, that reflect how the student experiences at the institution supported their development as individuals • Cognitive outcomes include demonstrable acquisition of specific knowledge and skills: • What do students know that they did not know before? • What can they do that they could not do before?

12. Accountability and Institutional Effectiveness • Major Players • American Association for Higher Education (AAHE) • Association of College and Research Libraries (ACRL) • Council for Higher Education Accreditation (CHEA) • United States Department of Education

13. Accountability and Institutional Effectiveness • Regional Higher Education Institutional Accreditation Associations: • develop standards by which to evaluate institutions • in the past, their standards have been inputs and outputs based, focusing on processes and structures concerning the delivery and use of library services. • to increase accountability, these bodies are placing more pressure on institutions to measure what students learn through assessment processes. • by focusing on results, rather than counting heads and library books, the regionals say, they are holding colleges accountable • want institutions to prepare assessment plans that measure outcomes within the institution.

14. Accountability and Institutional Effectiveness • Several regional accrediting agencies have revised their standards to include an emphasis on information literacy as a student learning outcome, and have identified participating roles for academic libraries. • NEASC - Standard Seven: Library and Information Resources • Standard 7.4 • Professionally qualified and numerically adequate staff administer the institution’s library, information resources, and services. The institution provides appropriate orientation and training for use of these resources, as well as instruction in basic information literacy

15. Four Perspectives (separate but interrelated?) • Library • Users: “customers” • Institutional • Broader community • Government (e.g., legislatures and departments of education) • National and regional accrediting bodies

16. Measures of Library Use • Inputs • the materials used to provide services such as collections, staffing, the physical facility and installed information technologies • usually described in financial terms • Outputs • measures quantifying workload undertaken and/or completed, such as the number of books circulated or the number of reference questions answered • valuable measures for making decisions about staffing levels, setting hours of operations, etc.

17. Measures of Library Use • It is important to track the library's inputs and outputs • libraries organize, track and evaluate measures by applying internal management information systems • report these measures via annual reports and surveys • Problems with library measures: • Librarians often misidentify outputs as outcomes • There is also confusion concerning definitions and applications • What is a login? What is a hit? Is a login a hit? • What is a page retrieval? An abstract or a page from a full text article? • Lack of standards leads to this confusion

18. Measures of Library Use • Input and output measures are limited: • If the mission of the University is teaching and research, how do these outputs measure that? • Outputs measure use of inputs – they do not measure results • Inputs and outputs are insufficient for assessing outcomes • Therefore, assessment is conducted to measure outcomes

19. Outcomes Assessment • Defined: change in ... • Local focus • Planning/ implementation/ analysis • Types of measures • Student outcomes • Student learning outcomes (information literacy or research)

20. Outcome Measures in Libraries • Outcome assessment • User oriented rather than institution centered • Focus on statements about what students will know/think/be able to do as a result of our programs. • Not statements about what the library should/could do to bring about desired outcomes. • Measures changes in library users as a result of their contact with an academic library’s programs, resources and services (but how do we know the impact results from the library?) • student known content, developed skills and abilities, and acquired attitudes and values

21. Outcome Measures in Libraries • measures the contributions the library made to the university or college’s educational mission as a whole • conducting student learning outcomes assessment is designed to improve library services.

22. Developing Partnerships with Faculty • Kenneth Smith’s report, New Roles and Responsibilities for the University Library: Advancing Student Learning Through Outcomes Assessment, states that the library must move from a content view (books, subject knowledge) to a competency view (what students will be able to do). • He contends that the student learning outcomes identified by ACRL are often common to learning outcomes identified by faculty for departmental courses and programs.

23. Developing Partnerships with Faculty • Therefore, the library could assist the student learning and assessment efforts of the academic programs by: • delivering “offerings” defined as “units of learning materials designed to develop competency in specific learning outcomes that are considered important by the library and by other academic programs.” • incorporate the offerings into required courses

24. Developing Partnerships with Faculty • The offerings are a means to “give the library a curriculum (its own set of course segments) and an opportunity to connect this curriculum to other academic programs.” • The library must internally develop these learning units and then proactively convince the faculty that the library can “contribute to the learning outcomes of the academic program,” because it is unlikely that the academic department will formally request such assistance from the library.

25. Direct Methods • Qualitative: developmental portfolios, think-aloud protocol, and directed conversations • Quantitative: content analysis, evaluation of theses/dissertations, tests (even ones administered as pre- and post- tests), videotape and audiotape evaluation, and nationally developed tests

26. Indirect Measures • Qualitative: focus group interviews, curriculum and syllabus evaluation, exit interviews, external reviewers, observation, self-assessment • Quantitative: general surveys

27. Example (Kathleen Dunn, p. 28) • Core Competency: “locate and retrieve information, in all of its various formats using, when appropriate, technological tools.” • Scenario: “You’ve been selected to lead a group of students on a four-week trip to another country. In preparation you need to find background information on thecountry’s history, current events, language, local customs, art, and music. How would you locate the information? What type of information would you expect to find? • How can we assessthe extent to which scenario met and change over time?

28. Another Example • Objective: to reduce the number of retrievals (hits) per search on aggregate and specialized databases • Is this objective measurable? What types of measures would you apply? • Can this objective be taught? How? • How would you measure if anything was learned? • What will the results tell you?

29. So… How should a library set assessment priorities?

30. Some Relevant Readings • An ActionPlan for Outcomes Assessment in Your Library (ALA, 2002) • The Journal of Academic Librarianship (Jan.-Mar. 2002); (Nov. 2002) • Outcomes Assessment in Higher Education (Libraries Unlimited, 2004) • Assessing for Learning, by Peggy Maki (Stylus, 2004) • Textbooks from Middle States Commission

31. Some Relevant Readings • Association of College and Research Libraries. Information Literacy Competency Standards for Higher Education. <http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm> • Association of College and Research Libraries. Standards for College Libraries 2000 Edition. <http://www.ala.org/ala/acrl/acrlstandards/standardscollegelibraries.htm>

32. Some Relevant Readings • Association of College and Research Libraries. Task Force on Academic Library Outcomes Assessment Report, June 27, 1998. • ERIC Clearinghouse on Assessment and Evaluation. <http://ericae.net/> • Smith, Kenneth R. New Roles and Responsibilities for the University Library: Advancing Student Learning Through Outcomes Assessment. <http://www.arl.org/stats/newmeas/outcomes/HEOsmith.html>

33. Management Information Systems (MISs) • Used to compile and report information concerning “how many” and “how much” to help libraries demonstrate accountability • Is an integrated reporting system specifically planned and designed to provide managers with specific information they need to perform their managerial functions effectively and efficiently

34. MIS • An MIS uses information that is internally produced: • Data generated by the integrated automated library system including counts (e.g., circulation) and reports (e.g., collection age report) • Other counts (e.g., number of reference questions received and of students attending instruction sessions) • Measurements (e.g., square feet) • User/use surveys (concerning user feedback)

35. MIS • An MIS uses information that is externally produced: • Institutional status reports (e.g., monthly expenditures) • Institutional reports (e.g., reports to governments) • Information from the library’s vendors (e.g., database use) • Demographics (e.g., number of students and program population)

36. Uses of a MIS • Planning • Use analysis • Cost analysis • Gap analysis • Information system for decision making • Compile numbers to determine status • As a management control mechanism • For evaluation (inputs and outputs) • Sawyer Library MIS

37. Uses of a MIS (for Benchmarking) • Ratios: how do we compare with others using the same measures? • ACRL Standards for College Libraries 2000 Edition “Points of Comparison;” http://www.ala.org/Content/NavigationMenu/ACRL/Standards_and_Guidelines/Standards_for_College_Libraries_2000_Edition.htm#points • Governors State University Implementation of Standards for College Libraries (2000 edition), http://www.govst.edu/uploadedFiles/GSU%20Implementation%202000(1).pdf • ARL Statistics & Measurement Program, http://www.arl.org/stats/ • Table of ratios used • http://www.rdugan.org/LIS451/mis/arl_stats_ratios.pdf

38. Statistics • ACRL 2003 Statistical Summaries • NCES: Compare Academic Libraries

39. Even More Uses • Compliance with standards • Regional accrediting bodies • NEASC Standard 7, Libraries and Information Resources, http://www.rdugan.org/LIS451/mis/NEASC_standard_7.pdf • Reporting to • Federal agencies (IPEDS survey instrument) • the Institution • Annual report, etc. • the library community • surveys from organizations, associations, and other libraries (ACRL Statistical Questionnaire)