The collegiate learning assessment cla
Download
1 / 33

The Collegiate Learning Assessment CLA - PowerPoint PPT Presentation


  • 525 Views
  • Updated On :

The Collegiate Learning Assessment (CLA). Roger Benjamin CAE April 15, 2008. Mission: Improve Teaching and Learning---Higher Order Skills. Strategy Initial institution-based comparison of student learning outcomes relative to the development of higher order skills

Related searches for The Collegiate Learning Assessment CLA

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Collegiate Learning Assessment CLA' - loc


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The collegiate learning assessment cla l.jpg

The Collegiate Learning Assessment (CLA)

Roger Benjamin

CAE

April 15, 2008


Mission improve teaching and learning higher order skills l.jpg
Mission: Improve Teaching and Learning---Higher Order Skills

  • Strategy

    • Initial institution-based comparison of student learning outcomes relative to the development of higher order skills

    • Encourage CLA blue print for continuous improvement

    • Launch Institute for Performance Assessment to advance role of performance assessment in education


Rationale l.jpg
Rationale

  • We do not have strong theories of teaching and learning

  • Comparative method useful to provide investigator with more cases to generalize from

    • Comparative based approach only makes sense if there is sufficient variation between colleges and universities on student learning outcomes


Collegiate learning assessment cla l.jpg
Collegiate Learning Assessment (CLA)

  • The performance tasks are designed to measure higher order skills, critical thinking, analytical reasoning, problem solving and writing initially at institution level of analysis.

  • CLA only measures certain aspects of undergraduate education.


Overview l.jpg
Overview

  • Principles driving the CLA

  • CLA’s distinguishing features

  • CLA’s measures

  • Reporting results

  • The CLA Blue Print for Continuous Improvement

  • The Case for an Institute of Performance Assessment


Principles driving the cla l.jpg
Principles Driving the CLA

  • Cannot assess everything that is important

  • Can measure some important abilities

  • Need benchmarks

  • Use results to support improvement and curricular reforms

  • Use psychometrically sound measures


1 one measure is not sufficient l.jpg
1. One measure is not sufficient

  • Colleges have different missions

  • Students have different majors

  • Multiple important outcomes require multiple measures


2 can measure some important abilities l.jpg
2.Can measure some important abilities

  • Competencies listed in college mission statements are applicable to all students

    • Writing – clear, organized, etc.

    • Critical thinking

    • Analytic reasoning

    • Problem solving

  • Integrated rather than separate abilities

  • Cut across academic majors


3 need benchmarks to assess progress l.jpg
3. Need benchmarks to assess progress

  • How much did our students improve from the time they entered college?

  • Is that degree of improvement comparable to that obtained by equally able students at other colleges?


5 use psychometrically sound tests l.jpg
5. Use psychometrically sound tests

  • Valid – assess important skills (and not something else)

  • Reliable – consistent (not chance)

  • Fair – standardized, calibrated, and unbiased measures

  • Cost effective in terms of testing time and expense


Some of cla s distinguishing features l.jpg
Some of CLA’s Distinguishing Features

  • Rely on open-ended holistic tests that are:

    • realistic “work samples”

    • engaging

    • applicable across academic majors

  • College (not student) is the unit of analysis

  • Sample students within schools

  • No arbitrary standards, such as “proficient”

  • Paperless administration, scoring, & reporting

  • Control for differences in input between schools

  • Report results in terms of “value-added”


Cla is not like nclb l.jpg
CLA Is Not Like NCLB

  • Constructed rather than selected response tests

  • Use assessment for improvement (low stakes) rather than accountability (high stakes)

  • Same tests used nationwide rather than a patchwork of measures and standards

  • Matrix sampling of measures (like NAEP) rather than require that all students take all measures

  • Test sample of students at each school

  • Schools and students volunteer to participate

  • Results released to the institution, not published


Cla s measures l.jpg
CLA’s Measures

  • Analytic writing (essay) prompts

    • Make-an-argument (45 minutes)

    • Break-an-argument (30 minutes)

  • Performance tasks (90 minutes)

    • Document based

    • Contextualized questions

    • Split screen/dialogue box format

    • Analytic and holistic scoring

  • Background questionnaire


Make an argument prompt l.jpg
Make-An-Argument Prompt

“In our time, specialists of all kinds are highly overrated. We need more generalists – people who can provide broad perspectives.”

Directions: 45 Minutes, agree or disagree and explain the reasons for your position. Student selects one of two prompts to answer.


Break an argument prompt l.jpg
Break-An-Argument Prompt

Students are asked to discuss how well reasoned they find an argument to be (rather than simply agreeing or disagreeing with it).

A respected professional journal with a readership that includes school principals published the results of a two-year study on childhood obesity. This study sampled 50 children, ages 5-11, from Jefferson School. A fast food restaurant opened near the school just before the study began. After two years, students who remained in the sample were more likely to be overweight—relative to the national average. Based on this study, the principal of Jones Elementary School decided to address her school’s obesity problem by opposing the opening of any fast food restaurants near her school.


Performance tasks l.jpg
Performance Tasks

  • 90-minute real life problems

  • General directions and context

  • Need to combine information from different types of sources

  • A few open-ended questions, no one “right” answer—must explain rationale

  • Split screen

    Right: Document Library

    Left: Question and answer block


Swiftaire 235 l.jpg
Swiftaire 235

  • You advise the president of DynaTech

  • DynaTech makes airplane navigation and communication equipment

  • DynaTech’s sales manager suggests buying a Swiftaire 235 to visit customers and demo its products

  • Recent accident – wing came off in flight

  • Student’s tasks

    • Review document library

    • Write memo discussing pros and cons

      of DynaTech getting a Swiftaire 235

    • Justify your recommendations


Document library l.jpg
Document Library

  • Newspaper article about recent accident

  • Transcript of interview about accidents

  • DynaTech email exchanges regarding reasons to buy a Swiftaire 235

  • Trade magazine article that compares Swiftaire 235’s performance and safety characteristics to similar planes

  • Manufacturer specifications and required pilot training for Swiftaire 180 and 235

  • Charts about airplane accidents and sales


Scoring rubric l.jpg
Scoring Rubric

  • Writing skills – clear, organized, persuasive

  • Analysis, problem solving, & reasoning skills

    • Integrates information from different sources

    • Recognize flaws and issues—not swayed by emotional arguments, faulty logic, irrelevant information, etc.

    • Anticipates consequences and implications of alternative solutions and strategies

    • See pros and cons of competing explanations, points of view, and arguments

    • Weighs evidence based on its credibility


Two definitions of value added l.jpg
Two Definitions of Value-Added

  • Do seniors score higher than freshmen with the same SAT scores?

  • Does a college’s seniors score higher than what would be expected given (1) their average SAT score and (2) the typical relationship between average SAT and CLA scores at other colleges?


Fig 3 relationship between mean act scores and mean total cla scores for freshmen and seniors l.jpg
Fig. 3: Relationship Between Mean ACT Scores and Mean Total CLA Scores for Freshmen and Seniors

31

27

CLAScore

23

19

15

15

19

23

27

31

ACT Score


Status of the cla 07 08 l.jpg
Status of the CLA, 07-08 CLA Scores for Freshmen and Seniors

  • 238 four-year institutions testing (roughly mirrors the Carnegie classification)

  • Community college and independent and suburban high school pilots underway

  • Harvard Medical School-based consortium

  • Feasibility study for OECD international version of CLA underway


The cla comparison strategy from the institution to the classroom l.jpg
The CLA Comparison Strategy: From the Institution to the Classroom

  • There is up to 3.0 standard deviation of difference between similarly situated institutions along the regression plot. These large differences provide faculty and administrators a signal about where their institution stands

  • This means there is a large canvas for studying best practices in the institutions that do better than predicted.


The cla blue print for continuous improvement l.jpg
The CLA Blue Print for Continuous Improvement Classroom

  • Correlate inputs, processes, and outputs with CLA results

  • Drill down to departments and programs to discover which are strong or weak contributors to their CLA results

  • Conduct audit of existing assessments

  • Examine best practices, e. g., more analytic based writing appears to help


Get published performance tasks into the classroom l.jpg
Get Published Performance Tasks Into The Classroom Classroom

  • Assist faculty to develop performance tasks that are based on the scoring guide of the published tasks

  • Choose case studies and problems for text material that is congruent with the documents in the performance tasks

  • Adopt a student-centered approach to teaching that calls for more analytic-based writing


The continuous improvement system l.jpg
The Continuous Improvement System Classroom

  • Do comparative value added institution-based testing with CLA

  • Carry out variety of responses, particularly those focused on the classroom

  • Re-test using CLA value added institution-based testing

  • Repeat the cycle


The evolution of the cla 1 l.jpg
The Evolution of the CLA (1) Classroom

  • Attract a sufficient number of institutions using the CLA protocol to corroborate the hypothesis there is wide variation between similar institutions in producing CLA results

  • Develop diagnostic logic necessary to “unpack” CLA results at institution level


The evolution of the cla 2 l.jpg
The Evolution of the CLA (2) Classroom

  • Develop resources useful for faculty and administrators to employ CLA blue print for improvement,

    • Analyze test results to provide more detailed information about strengths and weaknesses of students’ performance

    • Create CLAintheclassroom for classroom use

    • Launch CLA Academy workshops to teach faculty how to develop performance tasks

    • Design on line version of CLAintheclassroom


The kalamazoo story l.jpg
The Kalamazoo Story Classroom

  • The differences between academic divisions---humanities, social sciences, natural sciences appears to be a function of writing

    • Students who write well and who have had more experience making judgments abut the value of information appear to perform better on the CLA


Sample cabrini college faculty response l.jpg
Sample Cabrini College Faculty Response Classroom

  • If a faculty member is going to use this kind of assessment, they are going to rethink the way they teach because this type of assessment asks for students to pull together various forms of information and analyze it at a higher level.


The case for an institute of performance assessment l.jpg
The Case for an Institute of Performance Assessment Classroom

  • Knowledge economy requires stronger higher order skills. This development portends significant changes in teaching and learning,

    • Shift from lecture format to student-centered approach that emphasizes more analytic writing

    • Change in the balance of curricular and textbook focus from emphasis on content to case and problem-based approach that requires students to apply what they know to new situations

    • Change from multiple choice tests to open-ended essay tests better aligned with the other two goals


Rationale for institute l.jpg
Rationale for Institute Classroom

  • Lesson of human capital school---its time to focus transforming the content of the stock of knowledge and skills of our citizens

  • Multiple choice tests no longer sufficient

  • The field of performance assessment must be developed

  • Comparative advantage of CLA to launch the Institute


Frequently asked questions l.jpg
Frequently Asked Questions Classroom

  • Are not selective institutions penalized if value added growth is the metric?

  • What about absolute levels of performance compared to value added growth?

  • What is the methodological importance of establishing the base line characteristics of the students taking the CLA?

  • What about the motivation issue?


ad