Student Learning Outcomes: Stepping Stones to Academic Program Success Dr. Mia Alexander- Snow, PhD Day 2: Assessment Wo

1. Student Learning Outcomes: Stepping Stones to Academic Program SuccessDr. Mia Alexander-Snow, PhD Day 2: Assessment Workshop Florida Institute of Technology May 25, 2010

2. Workshop Agenda • Assessment Overview • Program Assessment Process • Program Assessment Strategy • College Competencies & General Education Assessment • Program Major Learning Goals (Objectives) • Outcome Statements: Think SMART • Assessment Methodology is MATURE • Data collection • “Closing the Loop” Program Improvement

3. Workshop Goals • Participants will • be able to identify the elements of a student learning outcome statement; • develop student learning outcome statements for a specific program and/or course; • be able to identify the elements of an assessment methodology for an outcome; • be introduced to curriculum and program mapping; • develop an assessment methodology for a program or course outcome; • select direct measures for outcomes which have the potential to provide useful data that lead to program improvement, and • learn about closing the outcomes assessment loop.

4. Assessment & Accreditation Overview

5. What is Assessment? Assessment is … the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving learning and Development (Palomba, C.A. & Banta, T.W. (1999). Assessment Essentials p. 4). The results gathered in the Assessment process are not to be used to: • impact the individual evaluation process • undermine or impinge on academic freedom or professional rights • replace or alter the collective bargaining process

6. Assessment is a Collective Process Key elements Cultural change Commitment from upper administration Use of existing organizational structures to implement planning and assessment Participation from all levels Long term commitment Capability to measure objectives related to student learning outcomes and related activities Dissemination of results and information

7. What is Accreditation? • Accreditation is. . . the procedure by which an authoritative body gives formal recognition that a body or person is competent to carry out specific tasks. In a certification system, an accreditation body will accredit or, in simpler language, approve a certification body as competent to carry out certification http://www.goldenwestcollege.edu/iec/assessment.html). • Accreditation has two fundamental purposes: • to assure the quality of the institution or program, and • to assist in the improvement of the institution or program. http://www.collegedegreeguide.com/articles-fr/accredited-college-university.htm

8. The Assessment Process supportsSACS Core Requirements and Comprehensive Standards

9. SACS Core Requirements Compliance is essential for gaining and maintaining accreditation with the Commission on Colleges. To maintain accreditation, an institution must meet all Core Requirements (2.1-2.11, including 2.12—QEP) 2.5 The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; and (3) demonstrate the institution is effectively accomplishing its mission. (Institutional Effectiveness) 2.7.3In each undergraduate degree program, the institution requires the successful completion of a general education component at the collegiate level that (1) is a substantial component of each undergraduate degree, (2) ensures breadth of knowledge, and (3) is based on a coherent rationale. For degree completion in associate programs, the component constitutes a minimum of 15 semester hours or the equivalent; for baccalaureate programs, a minimum of 30 semester hours or the equivalent. These credit hours are to be drawn from and include at least one course from each of the following areas: humanities/fine arts, social/behavioral sciences, and natural science/mathematic. 2.12 The institution has developed an acceptable Quality Enhancement Plan (QEP) that includes an institutional process for identifying key issues emerging from institutional assessment and focuses on learning outcomes and/or the environment supporting student learning and accomplishing the mission of the institution. (Quality Enhancement Plan)

10. SACS Comprehensive Standards The Comprehensive Standards are more specific to the operations of the institution, represent good practice in higher education, and establish a level of accomplishment expected of all member institutions. If an institution is judged to be significantly out of compliance with one or more of the Comprehensive Standards, its reaffirmation of accreditation may be denied.

11. SACS Comprehensive Standards 3.3 Institutional Effectiveness • 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: • 3.3.1.1 educational programs, to include student learning outcomes • 3.3.1.2 administrative support services • 3.3.1.3 educational support services • 3.3.1.4 research within its educational mission, if appropriate • 3.3.1.5 community/public service within its educational mission, if appropriate 3.3.12 The institution has developed a Quality Enhancement Plan that (1) demonstrates institutional capability for the initiation, implementation, and completion of the QEP; (2) includes broad-based involvement of institutional constituencies in the development and pro- posed implementation of the QEP; and (3) identifies goals and a plan to assess their achievement.

12. SACS Comprehensive Standards 3.4 Educational Programs: All Educational Programs (includes all on- campus, off-campus, and distance learning programs and course work) (See Commission policy “Distance and Correspondence Education.”) 3.4.1 The institution demonstrates that each educational program for which academic credit is awarded (a) is approved by the faculty and the administration, and (b) establishes and evaluates program and learning outcomes. (Approval of Faculty) 3.5 Educational Programs: Undergraduate 3.5.1 The institution identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies. (Competencies with General Education Core) Federal Requirements 4.1 When evaluating success with respect to student achievement in relation to the institution’s mission, the institution includes, as appropriate, consideration of course completion, state licensing examinations, and job placement rates. (Evaluation of Student Achievement)

13. The Assessment Process Supports Student Learning

14. Type Systemwide Assessment LevelCommunity college, 4-Year, or University System Goal To provide public accountability Type Institutional Assessment Level Individual College or university Goals To provide accountability; to gain and maintain accreditation Type Program Review and Assessment Level School, program, or department Goals To gain and maintain accreditation; to improve program quality and success TypeClassroom Research Level Individual course Goals To understand and improve teaching and learning in that individual course TypeClassroom Assessment Level Individual class meeting or assignment within the course Goals To understand and improve teaching and learning of a single lesson or unit

15. Developing an Effective Assessment Process

16. 9 Principles of Assessment Process(http://www.aahe.org/assessment/principl.htm) The Nine principles include: 1. The assessment of student learning begins with educational values. 2. Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time. 3. Assessment works best when the programs it seeks to improve have clear, explicitly stated purposes. 4. Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes. 5. Assessment works best when it is ongoing, not episodic. 6. Assessment fosters wider improvement when representatives from across the educational community are involved. 7. Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about. 8. Assessment is more likely to lead to improvement when it is part of a larger set of conditions that promote change. 9. Through assessment, educators meet responsibilities to students and to the public.

17. Developing an Effective Assessment Process The Assessment of Student learning is a participative, interative process that. . . • Provides data/information you need on your students’ learning. • Engages you and others in analyzing and using this data/information to confirm and improve teaching and learning • Produces evidence that students are learning the outcomes you intended • Guides you in making educational and institutional improvements • Evaluates whether changes made improve/impact student learning, and documents the learning and your efforts (The Higher Learning Commission ).

18. Implementing an Effective Assessment Program Strategy(Nichols 5-Step Model) • Program (unit) mission statement • Program (unit) desired outcomes or objectives • How many should be required ( 1, 2,3)? • What number & type of outcomes (Operational and/or SLOs)? • Measurement approaches • How many per outcome? (1, 2, 3)? • What type & number of methods is acceptable (1 indirect and 1 direct; 2 directs; 2 directs and 1 indirect)? • Measurement results • Planned use of results to improve program (unit) • Implemented quality improvements

19. Definitions Program Assessment: setting and assessing outcomes that will promote program improvement Types of outcomes: • Outcome: A specific, measurable statement that describes desired performance. • Operational outcome:A type of outcome that addresses operational or procedural tasks, such as functions, demand, resources, efficiencies, and satisfaction • Learning outcome/Student Learning Outcome (SLO):A type of outcome that describes • intended learning that students should meet on the way to attaining a particular degree, called Student Learning Outcomes. • intended learning that participants meet as a result of a unit’s program(s) or services(s), called learning outcomes

20. Steps 1-3: Defining Program Major Mission, Vision, Goals (Objectives), & Outcome Statements

21. Step 1: Defining Program Major Mission & Vision • Who are we? • Name of the office, department, or unit • What do we do? • Your unit’s primary purpose • For whom do we do it? • The stakeholders or customers of your unit

22. Step 1: Defining Program Major Mission & Vision Mission Statement • Brief (75 words or less) and comprehensive • It should make sense to someone who knows little or nothing about your unit. • It should rarely need revision and be able to endure changes in leadership. • It should lead to the development of goals, outcomes and performance measures for those outcomes.

23. Step 1: Defining Program Major Mission & Vision The mission structure: “The mission of (your office name) is to (your primary purpose) by providing (your primary functions or activities) to (your stakeholders).” (You may add additional clarifying statements.) *NOTE: the order of the pieces of the mission statement may vary from the above structure

24. Office Name Primary Stakeholders Primary Purpose Primary Functions Mission Statement Example The Mission of the UCF Office of Residence Life is to provide quality housing facilities and related services that are reasonably priced, safe, comfortable, well-maintained and staffed by friendly, caring, and efficient people to undergraduate students.The department develops and promotes programs and staff interactions that are conducive to student learning, support the University’s academic mission, and encourage individual responsibility within a community setting.

25. Step 1: Defining Program Major Mission & Vision Vision • What would you like your department to become? • Do you want to be the best at something and in what way? • What would you like to see the department be or do in the future? Vision Statement: The vision of (name of office) is to become (statement that will guide your future actions). Ask: • Is this a statement that will pull you in a desired direction? • Is your vision statement inspirational?

26. Step 1: Example of a Vision Statement The University of Central Florida’s Faculty Center for Teaching will foster the model environment where scholars innovate, invigorate, and explore the art of teaching and the science of learning.

27. Step 2: Defining Program Major Learning Goals (Objectives)

28. Step 2: Defining Program Major Learning Goals (Objectives) Describe an “ideal” student at various phases in your program, focusing on the abilities, knowledge, values and attitudes that you feel that this student has either acquired or have been supported as a result of your program. Ask: • Cognitive skills: What does the student know? • Performance skills: What can the student do? • Affective skills: What does the student care about? • Describe how the students’ experiences in the program have contributed to their abilities, knowledge, values and attitudes. • Describe the program alumni in terms of their achievements, such as career accomplishments, lifestyles, and community involvement. • List the skills and achievements expected of graduates of the program.

29. Step 2: Defining Program Major Learning Goals (Objectives) Collect and review any existing goal or outcome statements such as those from: • Catalog descriptions • Program review reports • Mission and vision statements • External agencies (e.g., Southern Association of Colleges and Schools Commission on Colleges [SACS - COC], National Council for Accreditation of Teacher Education [NCATE], Association to Advance Collegiate Schools of business [AACSB], and Accreditation Board for Engineering and Technology [ABET]) Review other programs’ goals • Review program goals and intended outcomes of other departments at your university. • Review program goals and intended outcomes of similar departments at other universities.

30. Steps 2Defining Program Learning Goals (Objectives) List five to seven of the most important goals identified in the sources. Prioritize the goals depending on their importance to your department and their universality (i.e., how well they apply to different program tracks, if applicable). Next, determine whether the goal is best described as knowledge, abilities, attitudes, values, or a combination of these. A goal can be described by more than one term. See the following illustration. (Adapted from Assessment Handbook, 2003)

31. Step 2: Example #1State Level CoreLearning Goals Florida State mandate (Academic Learning Compacts): Students who complete a baccalaureate degree in the State of Florida are expected to demonstrate the following Core competencies: All undergraduate programs at UCF develop 8–12 specific Student Learning Outcomes that address: • Communication : reading, speaking, writing, editing, questioning, listening, making presentations, and interpersonal relations. Some programs have additional modalities and/or techniques • Critical Thinking: outcome activities that require analysis, synthesis and evaluation • Discipline-specific knowledge, skills, attitudes & behaviors: vary by discipline and can vary by level, major or program Institutional links: All FL Universities: http://www.flbog.org/about/cod/asa/university_alc.php

32. Step 2: Example #2 Institution/College (General Education) Learning Goals (Objectives) • Students who complete a baccalaureate degree are expected to demonstrate the following competencies: • COMMUNICATION: demonstrate ability to engage in cross-communication styles—written, verbal, auditory, visual • QUANITATIVE : apply quantitative tools to solve real-world problems; • SCIENTIFIC: understand and apply relationships and theories of natural sciences; • SOCIOCULTURAL: demonstrate knowledge of the human condition and the importance of the arts; • GLOBAL SOCIOCULTURAL RESPONSIBILITY: Participate actively as • informed and responsible citizens in social, cultural, global, and • environmental matters • INFORMATION FLUENCY:Critically, gather, evaluate, and use technological information in an appropriate and effective way. • Adapted from College Core Competencies from South Plains College, Levelland, TX: http://www2.southplainscollege.edu/ & Florida Community College Statewide GenEd Requirements

33. Step 2: Example #3 College Competency/Program Major Learning Goals (Objectives) • CRITICAL THINKING: Observation, reflection, analysis, synthesis, application. • A graduate of ____ is able to… • Gather and analyze data relevant to a problem or position (ability). • Consider diverse perspectives and alternative points of view (values). • Predict and evaluate consequences and revise the thinking process (ability). • Select well-reasoned solutions and use sound evidence to justify a position (ability). • Apply knowledge to new situations and larger contexts (_______).

34. Step 2: Example #4 College Competency/Program Major Learning Goals (Objectives) • COMMUNICATION: demonstrate ability to engage in cross-communication styles—written, verbal, auditory, visual • A graduate of ____ is able to… • Comprehend and interpret various types of written information (knowledge). • Communicate thoughts, ideas, opinions, information, and messages in writing (ability). • Compose and create documents with correct grammar, spelling, and punctuation, as well as appropriate language, style, and format (_______).

35. Step 2: Example #5 College Competency/Program Major Learning Goals (Objectives) • INFORMATION LITERACY: Find, evaluate, organize, and use information. • A graduate of ____ is able to… • Locate and use appropriate and relevant information from print and/or electronic sources (knowledge, ability). • Identify, analyze, and evaluate information for relevancy, accuracy, authority, bias, currency, and coverage (_______). • Synthesize collected ideas and materials into original work in appropriate formats (knowledge, abilities). • Use information ethically and legally (values, ability). • Use appropriate technology to manage information, solve problems, communicate, develop products, and provide services (_______).

36. Step 3: Learning Outcomes Are they S.M.A.R.T.?

37. Step 3: Define Outcome Statements • Think S.M.A.R.T. • Specific • Clear and definite terms describing the abilities, knowledge, values, attitudes, and performance • Measurable • It is feasible to get the data; data are accurate and reliable; it can be assessed in more than one way • Aggressive and Attainable • The outcome has the potential to move the program or unit forward. • Results-oriented • Describe what standards are expected from students or aspect of the functional area being assessed • Time-bound • Describe a specified time period for accomplishing the outcome (e.g., by graduation, course completion) 38

38. Step 3: Define Outcome Statements Outcome Statement: A specific, measurable statement that describes desired performance. • More precise, specific, and measurable than a goal • Can be more than one outcomes related to each goal Types of outcomes: • Outcome: A specific, measurable statement that describes desired performance. • Operational outcome: A type of outcome that addresses operational or procedural tasks, such as functions, demand, resources, efficiencies, and satisfaction • Learning outcome/Student Learning Outcome (SLO): A type of outcome that describes • intended learning that students should meet on the way to attaining a particular degree, called Student Learning Outcomes. • intended learning that participants meet as a result of a unit’s program(s) or services(s), called learning outcomes

39. Student Learning Outcomes (SLOs) • Specific statements that describe required learning achievement that must be met on the way to attaining the degree or earning the certificate and meeting the goals of the program. • Ways statements can be framed: • What does the student know? (cognitive) • What can the student do? (psychomotor) • What does the student care about? (affective) 40

40. Step 3: Defining Outcome Statements--Learning Outcomes/Student Learning Outcomes (SLOs) Attention to Bloom’s Taxonomy • Cognitive: Developmental Process • Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation • Affective: Emotional Growth • Accepting, Responding, Valuing, Organize personal value system, Internalize value system • Psychomotor: Movement patterns, skills, and behaviors • Imitation, Manipulation, Develop precision, Articulation (combine, integrate related skills), Naturalization (automate, become expert)

41. Step 3: Example #1 Defining Student Learning Outcomes (SLOs) Speech Communications 1600/1016 Communication: Effective reading, writing, speaking, listening Outcome: Students who have completed SPC 1600 or SPC 1016 will understand and use extemporaneous delivery techniques to enhance information processing and persuasion of listeners. 42

42. Step 3: Example #2Defining Student Learning Outcomes (SLOs) Reading Education Communication: Effective reading, writing, speaking, listening Outcome: Students will demonstrate culturally responsive teaching strategies that meet the literacy needs of students in a diverse, multicultural classroom.

43. Step 3: Example #3Defining Student Learning Outcomes (SLOs) American History : 1877-Present Communication: Effective reading, writing, speaking, listening Outcome: Students will demonstrate knowledge of the chronology and significance of major events and movements in American history.       . 44

44. Step 3: Example #4Defining Student Learning Outcomes (SLOs) Art Communication: Effective reading, writing, speaking, listening Outcome: Students will demonstrate good visual literacy in their artwork (e.g., principles of design, technical applications, systems of perception, color applications, and conceptual issues).

45. Learning Outcomes: Rate the Examples Are they S.M.A.R.T.?

46. Learning Outcomes: Rate the Examples • Faculty will collaborate with M.Ed. Students to produce a successful conference proposal. • Tutors will demonstrate active listening techniques following the Summer 2008 training. • HVAC certificate students will accurately appraise systems likely to be present in specified building types. • After completing the A.S. program in Graphic Design, students will successfully redesign products according to customer specifications. • 5. Graduates of the Hypothetical Engineering BS program will recognize, integrate, and diagram essential components of engineering.

47. Learning Outcomes: Rate the Examples • Students will develop the competencies needed for teachers of social studies. • After completing SLS 1101, students will show an increase in their ability to use technological resources to conduct research. • HVAC certificate students will accurately appraise systems likely to be present in specified building types. • After completing the A.S. program in Graphic Design, students will successfully redesign products according to customer specifications.

48. Steps 4 -7: Assessment Methodology, Course & Curriculum (Program) Mapping

49. Step 4: Identifying Appropriate Assessment Methods • Consider these questions: • How will I know if this outcome is being accomplished? • What measures are currently available? • What method will provide me with this information? • What kind of assessment data: Formative or Summative?

50. Step 4: Identifying Appropriate Assessment Methods • Direct: direct examination or observation of knowledge, skills, attitudes or behaviors to evaluate the competence of students (participants) in the program • What does the student (participant) know? • What can the student (participant) do? • Indirect: concerned with documenting student (participant) experiences, opinions, or perceptions • What does the student (participant) report that he/she knows? • What does the student (participant) report he/she can do? 52