Voluntary system of accountability and learning outcomes an update
1 / 28

Voluntary System of accountability and learning outcomes: an update - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Voluntary System of accountability and learning outcomes: an update. Teri Hinds Voluntary System of Accountability Natasha Jankowski National Institute for Learning Outcomes Assessment. October 30, 2012 Assessment Institute. Overview NILOA’s work with Transparency

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Voluntary System of accountability and learning outcomes: an update

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Voluntary System of accountability and learning outcomes: an update

Teri Hinds

Voluntary System of Accountability

Natasha Jankowski

National Institute for Learning Outcomes Assessment

October 30, 2012Assessment Institute


  • NILOA’s work with Transparency

  • Internal and External Communication

  • Voluntary System of Accountability

  • Evaluation of VSA College Portrait Pilot

  • Future of Student Learning Outcomes on College Portrait

  • VSA as a communication tool

The Heat is On

  • Unprecedented public challenges for higher education to be more accountable for results.

  • Accompanying external demands for information about student and institutional performance are growing calls for institutions and accreditors to become more transparent about what they do and the results they achieve.

    (Ewell, 2010)

Overview of NILOA

NILOA’s mission is to document student learning outcomes assessment work, identify and disseminate best practices, and support institutions in their assessment efforts.

Surveys ● Web Scans ● Case Studies ● Focus Groups ● Occasional Papers ● Website ● Resources ● Newsletter ● Listserv ● Presentations ● Transparency Framework ● Featured Websites ● Accreditation Resources ● Assessment Event Calendar ● Assessment News ● Measuring Quality Inventory ● Policy Analysis ● Environmental Scan ● Degree Qualifications Profile



Reports on Transparency


  • What information does your institution share publically on its website?

  • How are results of student learning shared internally?

Transparency Framework


Transparent Communication

  • Internal communication

  • External communication

  • Telling an institutional story about student learning


  • What types of stories does your institution tell about its students?

  • What types of stories does your institution tell about how it assures quality?


  • VSA launched in 2007

  • Partnership between APLU and AASCU

  • 320 participating institutions

  • Three original objectives

    • Demonstrate accountability and transparency

    • College search tool

    • Support innovation in the measurement of student learning outcomes

Student Learning Outcomes Pilot

  • Student learning outcomes page includes link to institution specific learning outcomes data;

  • Selection of the CLA, CAAP or ETS Proficiency Profile to measure value-added learning gains

  • Report results no more than four years after joining

  • Link to information on test administration, sample and response rate

  • Pilot period ends 12/31/12

  • What was learned from the pilot?

Evaluate the Effectiveness of SLO Pilot

  • Focus groups

  • Interviews

  • Surveys

  • Google Analytics

  • College Portrait statistics

Findings related to Transparency

  • Both VSA-participating and nonparticipating institutions agree that VSA in 2007 was a wise, timely, useful, and necessary response to the accountability and transparency demands of the time

  • Yet, nearly half of the participants have not yet met expectations of College Portrait student learning outcomes pilot

    • 45% of participating institutions have yet to post student learning outcomes information

    • Information that was provided on student learning outcomes lacked explanatory or contextual material

  • 82% of participating institutions had traffic on their student learning pages but almost 1 in 5 did not have a single visitor to their student learning outcomes page since 2009.

  • Information posted on the College Portrait may not reflect the needs of prospective students and families or provide the information they actually seek to make decisions about where to attend college.

  • The authorized standardized test measures of student learning outcomes lack broad credibility and acceptance in the higher education community – it is thought the tests do not accurately reflect student performance within the institution.

  • Concerns about displaying results include:

    • Inability to understand what scores represent

    • Questions regarding what tests measure

    • Issues of reliability and validity

    • Difficulty in using scores to improve student learning

    • Motivation of students & difficulty to obtain a sample

    • Questions regarding quality of results

  • 81% of survey respondents stated that the pilot requirements did not align with their assessment practices and that the results were not usable for campus improvement efforts

  • Participating institutions agree - expanding the number & nature of student learning measures will:

    • produce more accurate portrayals of student attainment,

    • provide more useful information for campuses,

    • make information more meaningful for all audiences.


  • Focus on specific audiences and communicating meaningful information to them

  • Contextualize information

  • Expand range of assessment tools and approaches

  • Increase traffic to site

  • Include examples of use of evidence to improve learning

  • Consider creating template which could be used by all postsecondary institutions

Working Group Deliberations

  • Technical work group considered alternative measures of learning outcomes to recommend to VSA Board

  • Confirmed importance of student learning outcomes reporting within the VSA

  • No perfect of measure of students learning exists for all audiences

  • External accountability

  • Institutional improvement

  • College selection

Working Group Deliberations

  • Make value-add measurement using existing measures (CAP, CLA, ETS Proficiency Profiler) optional

  • Introduce option to use VALUE Rubrics based on AAC&U essential learning outcomes (written communication and critical thinking)

  • Introduce option to use aggregate scores from professional and graduate admissions exams such as the GRE, GMAT, LSAT, and MCAT

VSA Board Decisions

  • College Portrait re-focus from “college selection tool for prospective students” to “consumer information tool for institutions to demonstrate effectiveness of educational programs.”

    • Primary audiences: state officials, policy makers, consumers

  • Instrument options:

    • CAAP, CLA, ETS Proficiency Profile

    • AAC&U VALUE rubrics – critical thinking, written communication

    • GRE General Test

  • Reporting options: value-added, benchmarking

Implementation/Next Steps

  • All VSA institutions report results of pilot by end of 2012 to fulfill reporting requirement

    • 3 part report with tabbed display

      • Did you participate (or try to)? If so, how’d it go? If not, why not?

      • Give an example of another institutional-level evaluation

      • Which of the new options are you considering for the future?

    • Reporting Format Preview: http://www.collegeportraits.org/slo_preview.html

Implementation/Next Steps

  • Communications Task Force working with College Portrait Support Team on display of new options

    • Emphasis on more graphical displays

    • Additional guidance on what the scores and results mean

    • Click-through links for additional details, e.g., subscores

  • New options released for 2012-13 data cycle (Jan 2013)

  • Develop outreach strategy to recruit new participants and connect with key audiences – state officials, accreditors

More Information





Teri Hinds: thinds@aplu.org

Natasha Jankowski: njankow2@illinois.edu

Let’s Discuss.

  • Login