Program Learning Outcomes Assessment Tools Workshop

Program Learning Outcomes Assessment Tools Workshop Qatar University April 27, 2008 Facilitator Dr. K. El Hassan AUB.

Workshop Outline • Capstone Course • Course Embedded Assessment • Transcript Analysis • Value-Added Assessments • Portfolios

Focus • Definition • Advantages/disadvantages • Types • Creating and Designing Steps • Scoring/evaluating • Exemplars

Capstone Course: Definition • A capstone course is a course designed to be offered in the final semester of a student’s major, • It ties together the key learning objectives that faculty expect the student to have learned during the major, or interdisciplinary program • In small classrooms, students are expected to demonstrate the ability to integrate knowledge and to make interdisciplinary connections

Capstone Course Emphasis • Development: • Provide culminating experiences • Facilitate transition • Value reflection, synthesis, and planning for future. • Assessment: assess student learning outcomes at program level • What does a student know? • What can he do? • What evidence suggests that they know and can do? • Results used for instructional improvement, facilitating integration in the major and with general education purposes.

Capstone Course Advantages They enable: • Faculty to assess the cumulative abilities of students within the context of one course • Students to demonstrate how they can integrate the knowledge, abilities, and values

Capstone Disadvantages The capstone course may not: • allow enough time for students to devote enough time and effort to truly comprehensive projects. • produce the data faculty need if • the exercises or projects are not directly linked to the program learning outcomes, and • if the faculty teaching the course do not require what the program faculty have agreed upon

Types of Capstone Courses • The major project course. • students work on one project primarily, such as a research paper or an experiment or a creative project. • Students work on the project in stages, allowing faculty to determine students’ abilities to revise and/or reconceptualize their work. • Student presentations of the project may be both written and oral, • The multiple experiences or exercises course. • students must provide evidence through a variety of means, such as examinations, research papers, oral presentations, group work, and multimedia presentations. • The portfolio in the capstone course. • The major project may be a requirement that students produce a portfolio of work that includes a variety of evidence regarding students’ abilities. • The field experience or internship as a capstone course. • fulfilling a field experience or internship experience as the culminating activity in the program. • students can demonstrate their knowledge, skills, and values in a wide variety of ways. • may be evaluated by both a faculty member and a field supervisor under whom the student is working. • Evaluation is done using check sheets and evaluation forms that are completed both during and at the end of the experience, notes from advisory meetings with the student, and materials that the student produces during the experience.

Creating and Designing a Capstone Course • Determine the specific broad learning objectives for the academic program; • Determine how those are translated into the individual courses. • Departments then create matrices of department goals by courses that indicate which courses work on achieving which goals. • Determine the kinds of student work that should be expected during the capstone course. • Goals need to be translated into content standards and performance standards: that is the skills and knowledge the student should achieve and the level at which the student should demonstrate them. • Standards should be developed into rubrics

Course and Goals Matrix

Creating and Designing a Capstone Course • Design the capstone course to enable students to produce that work; • Determine how and when the faculty will assess the work that students produce; • Multiple evaluators are selected and they need to look for patterns of strengths and weaknesses across all the papers (or a sample of papers). • Inform students in the syllabus or related handouts how the objectives of the course are designed to reflect a culmination of their abilities, knowledge, and/or values. • The final step, then, is to use the data collected about the performance of the program to provide feedback to improve the major.

Keys to successful capstone experience • It is a culminating set of personal, academic, and professional experiences, accordingly, their primary focus should be on synthesis, integration, or application of the previously acquired knowledge rather on acquisition of new knowledge. • Types of experience depend on specific needs of discipline. • Should be situated near end of the program of study, and should be required for graduation. • Should be facilitated, mentored or coordinated by full-time faculty thus ensuring that general education themes and discipline specific requirements are addressed consistently. • Student ownership, responsibility and engagement are central with ample opportunities for reflection, discussion and demonstration of general education principles.

Evaluating the Work from a Capstone Course • Rubrics are used to evaluate major projects or portfolios • Use checklists or key questions when multiple evaluators are assessing. • Embed few questions in examinations • Professionals in field observe and evaluate student work • Faculty as a whole must review results • Results of evaluation must be looked at collectively.

Evaluating the Work from a Capstone Course • Check for gaps, weaknesses, compare with previous years, and see how programs are working. • The focused discussion among faculty members by itself is useful. ‘Taking a capstone course does not necessarily guarantee having a capstone experience, which is a well-thought out project that is comprehensive in nature and allows demonstration of a range of abilities. It also requires careful planning of activities by faculty responsible for the program not just course instructor.’

Exemplars: Organizational Leadership Capstone Occurs in two separate classes: • a professional development capstone in which students participate in a leadership assessment center, complete comprehensive exam drawn from core course content, and develop a reflection paper on general education. • Research component: propose a research topic, conduct literature review, collect and analyze data, interpret and summarize results, develop conclusions and recommendations and present findings in form of a thesis and multimedia presentation.

Exemplars: Women Studies Majors • Collect a portfolio of all materials from their Women Studies courses, • Write a short summary for each course. Specific questions about learning styles and experiences are provided to help guide their reflections. • They write a full paperabout their experience as a Women Studies major, tracing the main themes and content of their learning experience. • In addition, each student is asked to design and carry out a research project on a topic that extends their learning experience. Three faculty independently rate the reflective and research papers, using guidelines and specific questions about how the students' performance reflects each of the program's knowledge or skill goals.

Exemplars: Sociology • Adopts a holistic approach to assessing a sample of the capstone papers every other year. • A subset of faculty reading the papers consider the conceptual depth and clarity, use of resources, and the structure of the paper. • Multiple readers use indicators derived from the department’s goals to look for patterns and trends in student performance on a sample of papers from various years and then use that information to revise the sociology curriculum. • The readers rate each paper from on a five-point scale from fair to excellent on each indicator that is relevant for the paper.

Course Embedded Assessment : Definition • Departments collect assessment information or make use of already existing one in the classroom for program or institutional activities. • Process by which reviewers take a second look at materials generated by students in a course to see what evidence it reveals that students have met specified student learning outcomes. • It may also involve the design of new exam questions and/or assignments for the explicit purpose of providing group level information on the achievement of student learning outcomes associated with an academic major or the general education program. These questions are incorporated or embedded into final exams, research reports, and term papers in senior-level courses. • The student responses are then evaluated by two or more faculty to determine whether or not the students are achieving the prescribed educational goals and objectives of the department.

Course Embedded Assessment: Advantages • Motivation • Usefulness of scores • Relationship to objectives/outcomes • Flexibility/cost • Timeliness of Feedback

Course Embedded Assessment: Disadvantages • More complex assignments, such as research papers and projects, will have to be evaluated by a group of faculty using rubrics, thereby requiring more time; • Test scores in and of themselves will not provide satisfactory data; • Faculty teaching courses must include the embedded assessments that the program faculty decide upon.

Varieties of Embedded Assessments • Examinations. Specific questions can be inserted into specific examinations for the purpose of assessment. • Research Papers and Projects. Samples of student work from a variety of courses are evaluated by faculty or by external reviewers to see how well the students are meeting program knowledge and skills goals. These can be evaluated using a rubric. • Field Experiences or Internships. Student work produced as a result of the field work or internships can be used to assess their learning, such as logs, field notes, and observations.

Designing Embedded Assessments • Determine the specific broad learning objectives for the academic program; • If you have not already done so, determine how those are translated into the individual courses; • Conduct an inventory of the types of assignments given in the various courses; • Decide which assignments would serve assessment purposes as they are and which might have to be modified to accommodate the assessment; • Integrate the embedded assessments within the courses; • Specify assessment criteria. • Devise a way to gather the results of the assessments and translate those results for the entire faculty; • Determine strengths and weaknesses of the students as a result of the assessments; • Make appropriate changes to the curriculum.

Development of Course Embedded Questions • Specify the educational objective/intended outcome. • Develop a set of questions related to the objective/intended outcome. • Select an appropriate evaluation instrument/exercise (examination, lab assignment, other written exercise, etc.). • Embed questions in the instrument/exercise

Notes for development of questions • Questions embedded in classroom exams and assignments are often developed by faculty committees, providing assessment beyond that devised for the particular course by its instructor(s). • Student papers and portfolios and performance on embedded questions are independently evaluated by more than one faculty member, at least one of whom is not an instructor in the course from which the material came. • In some units, a separate outcomes assessment committee or undergraduate curriculum committee is responsible for these evaluations

Exemplars: Classics • Classics faculty review students' translations of selected passages in the final exams of classes at various levels, from beginning courses to advanced ones. The students have not seen these passages before. The translations are evaluated by at least two faculty members, one of whom is the course instructor.

Exemplars: Art History • A sample of senior level art history papers are collected from graduating Fine Arts majors and presented to outside evaluators. Each year's evaluators write comments about the overall quality of the sample, then select twelve papers and discuss them in-depth as a team. They rate the papers on a five-point scale for each relevant program goal, and submit individual evaluations for each student and a final written report summarizing their findings.

Exemplar Assessment Plan: Biology Intended Outcome • Students completing senior level biology coursework will demonstrate expertise in using standard scientific format to prepare reports. Related Course(s) • BIOL 404 Assessment Measure(s) and Technique(s) • As a course requirement, BIOL 404 students will prepare weekly lab reports. At least two members of the program assessment committee will evaluate randomly selected reports using a checklist for determining students’ use of the appropriate format. Assessment Criteria • Evaluators will cite appropriate use of standard scientific format in over 80% of randomly selected student lab reports

Transcript Analysis Using student database to • examine course-taking or grade patterns, • what courses students take and in what order, • patterns in student grades, grade point averages of different specializations, • Identify areas of growing and waning interest • together with exit interviews, provide justification for development of new courses or curriculum revision. • assist advisors to better help their students in planning

Transcript Analysis • Allows more complete picture of students, • are majors who follow a particular course-taking path more likely to succeed? • Do department introductory courses attract students to the major program? • Are entering qualifications sufficient? Look at students with various entering qualifications and their performance and accordingly decide on qualifications.

Value-Added Assessment: Definition • Value-added assessment (or pre-post assessment) attempts to measure student growth over time, from the time that a student enters a program until the student graduates. The most common method is pre- and post- testing, although other types of evidence could conceivably be developed.

Value-Added Assessment: Advantages • Assessing the students when they first enter a program can establish a firm benchmark against which to measure growth or value-added. • Pre-testing is especially helpful for measuring student knowledge, or cognitive learning, and skills, though somewhat less so for measuring values. • Pre- and post- testing may work best with traditional four-year undergraduates rather than the more common situation now where students enter, stop-out, transfer, return, and take six years or more to graduate. • Pre- and post- testing can be easily scored. • Pre- and post- testing can be relatively easily analyzed using statistical procedures.

Portfolios: Definition • A systematic collection of student work, usually representing student work over time, such as from the first course in the major until the last semester of the senior year. • Collections of student work that exhibit student’s progress and achievement in given areas. • Included may be research papers, multiple choice/essay exams, self-evaluations, personal essays, journals, problem, cases, audiotapes, etc. • Information about students’ skills, knowledge, development, quality of writing, critical thinking can be acquired through comprehensive collection of work samples. • Can be assembled in a course/sequence of courses. • Evaluation done based on educational goals and objectives. Collecting information over time provides opportunity for assessing progression in acquiring a variety of learning objectives

Portfolios: Advantages Enable faculty to • assess complex sets of tasks and objectives, with examples of many different types of student work, including interdisciplinary learning and capabilities; • assess more rigorous and higher order thinking, such as application, synthesis, and evaluation; • track student work over time; • examine not only final student projects, but also, if the faculty think it worthwhile, to look at drafts and earlier phases of student projects; • place the responsibility for demonstrating competence or mastery upon the student; • help students reflect upon their learning and, in the process of compiling the portfolio, to understand more about what they have and have not yet learned; • provide students with documentation for job applications or applications to graduate school.

Portfolios: Disadvantages They • Require more time for faculty to evaluate than tests or single-sample assessments; • Require students to compile their own work, usually outside of class; • Do not easily demonstrate lower-level thinking, such as recall of knowledge; • May threaten students who limit their learning to cramming for tests or doing work at the last minute; • Require a system of storage that may take time or space to set up

Varieties of Portfolios • Electronic portfolios. • Showcase portfolios • Comprehensive portfolios • Open-ended portfolios

Creating a Portfolio Assessment System • Determine the specific broad learning objectives for the academic program; • List the kinds of student work that students might include to demonstrate mastery of the learning outcomes; • Determine which kind of portfolio you want students to create; • Develop a rubric to score the portfolio (see below); • Include the rubric with your instructions to students so that they understand how the portfolio will be evaluated; • Write instructions for the students on how to create the portfolio and how it will be used;

Creating a Portfolio Assessment System 7. Inform students that they are responsible for creating the portfolio; 8. Instruct students to label each part of the portfolio according to the learning objective being demonstrated; 9. Instruct students what they are to write as an introduction to the portfolio or as a reflective essay; 10. Instruct students that they are to discuss each sample of student work included in the portfolio, either as an introduction to the sample or within the introductory or reflective essay; 11. Determine how and when students will first be introduced to the portfolio requirement, such as an introductory course in the major or when the student meets with his or her advisor; 12. Score the portfolio using the rubric that you devised

Portfolio Planning Sheet*. Establish purposes of portfolios for students Identify how portfolios will affect students (required? graded?) Identify what students should demonstrate/learn from portfolios Articulate particular learning goals that will be addressed Determine content of portfolios Identify types of items (writing samples, exhibits, exercises, and so on) Establish selection criteria for items Describe reflective statements that need to be included Provide opportunities for feedback to students Create review teams Establish review cycle Provide other opportunities for consultation Establish scoring approach Develop specific scoring criteria and rating scale Develop a reliable process for examining and rating portfolios Provide for training of raters Establish procedures for programmatic assessment Decide whether to review all portfolios or a sample Create process for summarizing results Allow for faculty discussion of results Provide opportunities for developing recommendations Practical considerations Provide a timeline for portfolios activities and review Provide for storage of materials

Scoring a Portfolio: Many possibilities. • Simple checklist to check all items re included, then • Scoring rubrics to evaluate appropriateness and quality of items. • Some are scored holistically, collection is examined and given one overall grade (ex, average, unsatisfactory) based n how well it demonstrates mastery of objectives of program, • It can be scored separately on various criteria such as completeness, conciseness, organization, variety of materials, and professionalism. • Multiple trait scoring has been recommended that includes dimensions of what captures essential quality of work. • Each item can be scored, rather than overall portfolio, and then separate criteria can be developed for different types of entries.

Scoring a Portfolio: Many possibilities • Example: CriteriaRating ScaleScore Organization 3 has clear framework 2 framework present, but lacks clarity 1 framework not apparent Completeness 3 All elements present 2 Most items present 1 Several items present • Reliability and consistency of results is necessary, need for agreement among evaluators. • An integrative approach to scoring is needed, evaluators need to work together to construct a coherent interpretation, use faculty from other programs

The Scoring Rubric • List each learning objective in which the student must demonstrate mastery. • Develop a scale for scoring the rubric, such as one of the options below:a. Grades for each learning objective;b. A score (e.g., 1 – 5) for each learning objective;c. A scale of three categories, e.g., below standard, meets the standard, and exemplary. • Review actual portfolios to establish common agreement about the scoring. The faculty should agree upon what constitutes a specific grade, score, or category, perhaps reading a sample of portfolios together and discussing them before scoring all portfolios. • Allow some space for faculty to make some notes on their observations, so that the team assessing the portfolios can summarize the strengths and weaknesses that faculty can address later.

Example of Scoring Rubric (b)

Example of Scoring Rubric (c)

Portfolios Use for Program Evaluation • Faculty need to have some agreement about how • the portfolios will be viewed collectively, • how results will be summarized in a meaningful way and conclusions drawn for the whole program. • Decide on how many portfolios to examine, take a sample, and • Discuss what results imply about program, its strengths and strengths weaknesses.

Exemplar University of Toledo. • Report prepared to department faculty by committee on their evaluation of portfolios of all majors graduating in the preceding academic year. • Report included data on average and range of scores for each of the papers in portfolio, average and ranges of improvement from first paper to last, and average scores in each of five areas of evaluation with comment on particularly weak and strong areas. • Appropriate action was recommended for areas of concern. • This information and from other sources will be used to make decisions about revisions in courses or curriculum requirements, or revision in instructional methodology

Thank You kelhasan@aub.edu.lb

Program Learning Outcomes Assessment Tools Workshop