outcomes assessment linking learning assessment and program improvement l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
OUTCOMES ASSESSMENT: Linking Learning, Assessment and Program Improvement PowerPoint Presentation
Download Presentation
OUTCOMES ASSESSMENT: Linking Learning, Assessment and Program Improvement

Loading in 2 Seconds...

play fullscreen
1 / 45

OUTCOMES ASSESSMENT: Linking Learning, Assessment and Program Improvement - PowerPoint PPT Presentation


  • 256 Views
  • Uploaded on

OUTCOMES ASSESSMENT: Linking Learning, Assessment and Program Improvement. James O. Carey Associate Professor Emeritus School of Information University of South Florida. ALA Annual Meeting, June 27, 2011. Overview. What is outcomes assessment? Why do outcomes assessment?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'OUTCOMES ASSESSMENT: Linking Learning, Assessment and Program Improvement' - pier


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
outcomes assessment linking learning assessment and program improvement

OUTCOMES ASSESSMENT:Linking Learning, Assessmentand Program Improvement

James O. Carey

Associate Professor Emeritus

School of Information

University of South Florida

ALA Annual Meeting, June 27, 2011

overview
Overview
  • What is outcomes assessment?
  • Why do outcomes assessment?
  • What are the required elements of outcomes assessment?
  • How is an outcomes assessment process implemented?
  • Practical tips for successful outcomes assessment
  • Summary and Conclusions
what is outcomes assessment
What is Outcomes Assessment?

3 Characteristics

  • Identifying desired outcomes
  • Assessing progress on outcomes
  • Using the results of assessment for improvement

Roughly = To:

  • Institutional effectiveness
  • Accountability
  • Continuous improvement
  • Quality assurance
  • Formative evaluation
what is outcomes assessment4

NEEDS PLANNING

ASSESSMENT PROGRAM

What is Outcomes Assessment?

Continuous process . . .

. . . instead of an event.

what is outcomes assessment5
What is Outcomes Assessment?
  • At the institutional level
    • Institutional effectiveness
    • Accountability
  • For example:
    • The University’s goal is to graduate 80% of entering freshmen in 5 years
    • A three-year assessment indicates graduation rate of 63% in 5 years and points to financial problems as primary cause.
    • Office of Student Affairs will create a student services taskforce on alternative paths to financial viability
what is outcomes assessment6
What is Outcomes Assessment?
  • At the program level
    • Accountability
    • Outcomes assessment
  • For example:
    • A departmental objective is relevant job placement for 85% of graduates within 1 year of graduation
    • Surveys of alumni indicate 70% have found relevant placements in first year
    • An initiative is planned to contact relevant professional constituencies and develop structured involvement in the program with regional employers
what is outcomes assessment7
What is Outcomes Assessment?
  • At the program level
    • Student Learning outcomes assessment
  • Another example:
    • A library school wants graduates to be able to write an action research plan for a given problem scenario.
    • On the comprehensive exam, 18% of students fail the action research question and analysis indicates that most took the course with an adjunct
    • The curriculum committee standardizes the syllabus for the research course and the pass rate increases
what is outcomes assessment8
What is Outcomes Assessment?
  • At the classroom level
    • Student learning outcomes assessment
  • For example:
    • The instructor wants students to learn how to analyze service needs for a specified patron group
    • On the final project students consistently confuse patron needs with programming alternatives
    • The instructor develops new case studies on analyzing service needs, and performance on the final project improves to an acceptable level
what is outcomes assessment9
What is Outcomes Assessment?

To summarize:

  • Has three basic components
  • Is part of a whole family of accountability methodologies
  • Goes by many names
  • Is used in many organizations in both public and private sectors
  • Is used at many levels within organizations for multiple accountability purposes
our purpose today
Our Purpose Today
  • Institutional and programmatic learning outcomes assessment
    • For accreditation
    • Mandated, unit-level accountability
    • Shared responsibility

Rather than

  • Course-level outcomes assessment
    • For course improvement
    • Elective, individual accountability
    • Personal responsibility
our purpose today11
Our Purpose Today

Focus on:

  • Program level accreditation requirements
  • Systematic planning and evaluation
  • Student learning outcomes assessment
why do outcomes assessment
Why Do Outcomes Assessment?
  • A systematic process of outcomes assessment is currently required for accreditation by:
    • ALA Committee on Accreditation
    • Parallel professional association accreditation (e.g., NCATE)
    • All eight regional higher education accreditation organizations
    • Most state boards of regents and departments of education
  • A proven methodology for best results from effort and resources
student learning outcomes
Student Learning Outcomes
  • What are they?
    • Statements describing knowledge and skills that students are expected to master by completion of their program of studies
  • Synonyms (sort of):
    • Core competencies
    • Learning objectives
  • Where do we get them?
    • Parent institution’s mission, goals, and strategic objectives
    • Unit-level mission, goals, and objectives
student learning outcomes16
Student Learning Outcomes
  • Where do we get them? (continued)
    • Professional standards
      • ALA/COA Standards for Accreditation (2008) (see Standards I and II)
      • ALA Task Force “Core Competencies” (2009)
      • ALA divisions and other library/information professional associations
    • Expert faculty members
    • Syllabi from core courses
    • Program advisory boards, alumni, employers, practitioners, students
    • Exemplary LIS programs
    • Futurists
student learning outcomes17
Student Learning Outcomes
  • What do they look like?
    • Declarative sentences describing what students will know and be able to do
    • Description of a single skill or set of closely related skills that can be assessed at the same time
    • Performance of the skill(s) should be observable, or result in an observable product
    • Performance of the skill(s) should be measurable; i.e., one can determine when it has been done successfully
student learning outcomes18
Student Learning Outcomes
  • How many should we have?
    • Don’t go overboard!
    • Remember, if you write it you will need to assess it and report on it.
    • Usually 2-5 outcomes for each core content area are sufficient.
      • Write outcomes at a high intellectual level (analysis and problem solving) that subsumes multiple sub-skills
      • For example:

Students use strategic planning processes to guide the direction and progress of an organization VS

Students list the steps in a strategic planning process

student learning outcomes19
Student Learning Outcomes

Bad Ones:

Good Ones:

Students join relevant professional organizations.(observable &measurable)

Students plan a simulated needs assessment for a given collection development problem(observable and measurable)

  • Students appreciate the value of professional organizations
  • Students become familiar with needs assessment for collection development
student learning outcomes20
Student Learning Outcomes

Bad Ones:

Good Ones:

Students select scholarly literature appropriate for analyzing a current issue in LIS(observable and measurable)

Students describe principles of fair use and write policy for applications in an information center(observable and measurable)

  • Students know the scholarly literature in the LIS field
  • Students understand principles of fair use and how to apply them
student learning outcomes21
Student Learning Outcomes

Bad Ones:

Good Ones:

Students select a source of outside funding and write a proposal for support of a project(higher level skill)

?

  • Students find sources of outside funding for libraries and information centers
  • Students learn about cataloging tools and bibliographic utilities
student learning outcomes22
Student Learning Outcomes

Bad Ones:

Good Ones:

?

?

  • Students list the features of an effective reference interview
  • Students describe functional areas within libraries or information centers that offer opportunities for applied research
develop measures of outcomes
Develop Measures of Outcomes
  • IF learning outcomes have been written well, logical measures are often implied
  • For example:

Outcome: Students will identify and assess the specific information needs of user groups in the community and use that information to write a collection development policy

Measure: Write a collection development policy for user groups in the following community scenario

develop measures of outcomes25
Develop Measures of Outcomes
  • Characteristics of good measures
    • Valid; that is, actually measures what it claims to measure
    • Reliable; that is, will yield consistent scores
    • Applied uniformly across all students or sampled across students
    • Objective or require consensus of more than one judge/evaluator/rater
develop measures of outcomes26
Develop Measures of Outcomes
  • Two general types of measures:
    • Direct measures (primary data)

Describe the selection and configuration of technological resources required to solve the communications problems depicted in the following case study. (requires a product)

    • Indirect measures (supplemental data)

How would you rate your ability to select and configure technological resources to solve communications problems?

        • Not adequate for an entry-level professional
        • Adequate for an entry-level professional
        • Above average for an entry-level professional
        • Equal to an experienced professional

(elicits an opinion)

develop measures of outcomes27
Develop Measures of Outcomes
  • Examples of direct measures
    • Comprehensive examination w/rubric
    • Portfolio w/rubric
    • Products from capstone course w/rubric
    • Observation scale from fieldwork or internship
    • Standardized tests (local, state, or national)
    • Common course examinations
    • Licensure examinations
  • All examples must conform to characteristics of good measures
develop measures of outcomes28
Develop Measures of Outcomes
  • Examples of indirect measures
    • Exit interviews
    • Focus groups with students, alumni, supervisors, and employers
    • Surveys of students, alumni, supervisors, and employers
    • Reviews by advisory boards or councils
    • Case studies of cohort groups
develop measures of outcomes29
Develop Measures of Outcomes
  • What about students’ grades in classes, seminars, capstone courses, fieldworks, and internships?

NO WAY!

  • Can not satisfy the characteristics of good measures
develop measures of outcomes30
Develop Measures of Outcomes
  • Set performance expectations
    • Once measures have been established, set the levels of performance that will be considered acceptable or “passing”
    • This is an internal “gatekeeping” function for student progress
    • for example:
      • Yes or no; right or wrong; pass or fail
      • 80% correct, 90% correct, 95% correct
      • Average rating of 4 on a 5 point scale to pass
      • Students must perform at 4.5 level on critical criteria #1 and #2, but can pass with an overall rating of 4.0 averaged across all 5 criteria
develop measures of outcomes31
Develop Measures of Outcomes
  • Practical considerations
    • Not a simple task
    • Requires a level of sophistication in testing and measurement
    • Requires a committee of the willing and/or a layer of administration for:
      • Design and development of direct measures
      • Design and development of indirect measures
      • Formative testing and revision of both direct and indirect measures
assess learning outcomes
Assess Learning Outcomes
  • Developing and carrying out an assessment plan
  • Practical considerations
    • Requires a committee of the willing and/or a layer of administration for:
      • Policy, procedures, and calendar for administration of measures
      • Policy, procedures, and calendar for grading
      • Policy, procedures, and calendar for notifying successful students and notifying and managing unsuccessful students
      • Procedures and calendar for recording, summarizing, and reporting results
assess learning outcomes34
Assess Learning Outcomes
  • More practical considerations
    • It is difficult to measure all student learning outcomes with a single instrument in a single event
    • Use multiple measures, for example:
      • Comprehensive exam and products from capstone course
      • Capstone course products and portfolio
      • Portfolio and fieldwork observations
    • Sample across outcomes and students, for example:
      • Measure several outcomes in each comprehensive exam and rotate outcomes across exams each semester
organize and interpret results
Organize and Interpret Results
  • Purposes for organizing and interpreting results
    • Confirm satisfactory performance
    • Detect performance problems
    • Detect faulty assessment instruments and/or procedures
    • Discover opportunities for programmatic expansion, reorganization, additions, cuts, and changes in overall direction
    • Address accountability expectations for the parent institution and for accreditation
    • Inform programmatic improvement
organize and interpret results37
Organize and Interpret Results
  • Set accountability expectations
  • How do we know when the program is meeting its obligations to its students, its institution, and its profession?
  • For example:
      • 90% of alumni will report “adequate” or better preparation on 90% of learning outcomes
      • 85% of our students will achieve an average rating of 4.5 or above on a 5-point scale on their capstone projects
      • 95% of students will rate a “pass” on student learning outcome #6 on the comps exam
organize and interpret results38
Organize and Interpret Results
  • Methods for organizing and interpreting results
    • Matrix analysis is most typical
    • Display student learning outcomes by measurement items and fill in results at the intersection, for example:
use results for improvement
Use Results for Improvement
  • Remember—no need to improve what is working well!!!
  • The dependent variable in this investigation is student learning
  • The independent variables are many!
    • Teaching and learning are parts of a complex system with multiple interacting components
    • Data can point to problems, but cause and effect relationships are difficult to establish
simplified model of variables in teaching and learning see figure 1 in handout

COURSE

CONTENT

LEARNER

CHARACTERISTICS

LEARNING

ENVIRONMENT

STUDENT LEARNING

OUTCOMES

ASSESSMENT

IMPROVEMENT

IMPROVEMENT

  • ESSENTIALS FOR
  • LEARNING:
  • MOTIVATION
  • LEARNINGGUIDANCE
  • ACTIVESTUDENTPARTICIPATION
  • CONTENTINTEGRATION
Simplified Model of Variables in Teaching and Learning(See Figure 1 in handout.)
use results for improvement42
Use Results for Improvement
  • Where would one look among all of the variables for opportunities for improving student performance?
    • Begin with assessment data
    • Look for gaps between performance expectations and actual performance
    • Look for gaps between accountability expectations and actual performance
    • Sharpen understanding of performance problems with qualitative data

(See Table 1 in handout.)

practical tips for implementation
Practical Tips for Implementation
  • Establish authority for outcomes assessment

A person or committee charged with managing the process and delegating assessment responsibilities

  • Establish an annual assessment calendar

(See Table 2 in handout.)

  • Establish a uniform reporting format

(See Table 3 in handout.)

summary and conclusions
Summary and Conclusions
  • For good teachers, learning outcomes assessment is intuitive; good teachers are always improving what they do based on the results of what they have done in the past.
  • The challenges:
    • Infuse the logic of that “good teacher” intuition school-wide or department-wide
    • Create sustaining policies and administrative structures
    • “How we do it.” instead of “What we do for accreditation.”