Learning Outcomes Assessment in Student Services Methods, Strategies, Resources, and Considerations Jerry Rudmann Coas

1. 1 Learning Outcomes Assessment in Student ServicesMethods, Strategies, Resources, and ConsiderationsJerry RudmannCoastline College714-241-6338jrudmann@mail.ccc.cccd.edu

2. 2 Resources to help you at your campus Standard IIB – available from the AACJC In this PowerPoint: Discussion Questions – for getting people thinking about learning outcomes in Student Services Examples of assessment tools Criteria for evaluating assessment tools Sources of SLOs in Student Services Worksheets for individual and group work (in a MS Word document)

3. 3 Content review of standard IIB Learning support needs of students Identify, support, assure, evidence Learning outcomes specified in IIB Personal and civic responsibility, intellectual, aesthetic, and personal development Understanding and appreciation of diversity Contribution to student learning outcomes Evidence

4. 4 Discussion and Thought Questions We wish to acknowledge and thank Kathe German of City College San Francisco for contributing these excellent questions.

5. 5 What are some major outcomes we expect students to achieve as a result of a college education? Develop self-aware and interpersonally sensitive individuals Developing democratic citizens who participate in the political and social workings of the society Becoming persons having a master of a body of knowledge and insight into oneself and the world Development of skills that would qualify one as a skilled worker Development of lifelong learning skills

6. 6 What is the purpose of student learning outcomes and why are they important (other than to meet the new accreditation standards)? To provide a focus, both for staff and students – a way of articulating the primary purpose for the institution. To provide a framework against which to assess, document, and improve the achievement of learning outcome. Help focus our efforts on student growth, and thereby influencing the structure and delivery of programs and services. Help our constituencies, from students to the public, understand our intentions, affirm the values of our contributions, and document the impact Student Services is/are having on students.

7. 7 What do student learning outcomes have to do you with, as a faculty member, staff person, or administrator working in Student Services? We – you and I – are the purveyors of these experiences, and we – you and I – are the only ones who can really identify and foster the development of student learning outcomes within Student Services at our own college.

8. 8 What Student Service functions/units are represented by those in the room today? Go around and record on the board or chart.Go around and record on the board or chart.

9. 9 Can any of you provide some possible learning outcomes that your Student Services function impacts? That is – what are ways in which you already address learning outcomes with your students? Please give a brief example. E.g., the DSP&S counselor who, after some thought, said that one of the most important outcomes for her students is that they learn to advocate for themselves when interacting with faculty and staff. She then began thinking and listing ways in which the important outcome could be assessed (how and when).Please give a brief example. E.g., the DSP&S counselor who, after some thought, said that one of the most important outcomes for her students is that they learn to advocate for themselves when interacting with faculty and staff. She then began thinking and listing ways in which the important outcome could be assessed (how and when).

10. 10 What are some challenges and opportunities that we face as we try to address identification and assessment of learning outcomes? Commuter college Bureaucratic inertia Shrinking resources and tight budgets More… Start two lists. Don’t forget opportunities – e.g., to bring attention and well-deserved recognition to the fine work done by Student Service units.Start two lists. Don’t forget opportunities – e.g., to bring attention and well-deserved recognition to the fine work done by Student Service units.

11. 11 What impact will the implementation of student development and learning outcomes have on your role and recognition within the institution?

12. 12 Examples of some useful assessment tools

13. 13 Instruments Home grown survey instruments CCSEQ (Community College Student Experiences Questionnaire) The LASSI (Learning & Study Strategies Inventory)

14. 14 Home-grown surveys Coastline developed a student service survey to assess, for each service, the following… Have you heard of the service? If you have heard of the service, have you used it? If you have used the service, please express your level of satisfaction with the service. Please comment on the service. This survey is administered to classes. Online version to selected web classes Hardcopy version to selected classroom courses To student who file a petition to graduate

15. 15 College Activity Areas on the CCSEQ (83 items) Course Activities Library Activities Faculty Student Acquaintances Art, Music, Theater Activities Writing Activities Science Activities Vocational Skills Clubs and Organizations Athletic Activities Counseling and Career Planning Learning and Study Skills Each of these represents a specific area and the student responds to questions about how often he/she engaged in the activity during the current school year. EXAMPLES: Library: Prepared a bibliography or set of references for a term paper or project. ART, Music Theater: Attend a concert or other musical event at the college Clubs/Organizations: Looked in the student newspaper or on bulletin boards for notices about campus events and student organizations. Attended a meeting of a student club or organization.Each of these represents a specific area and the student responds to questions about how often he/she engaged in the activity during the current school year. EXAMPLES: Library: Prepared a bibliography or set of references for a term paper or project. ART, Music Theater: Attend a concert or other musical event at the college Clubs/Organizations: Looked in the student newspaper or on bulletin boards for notices about campus events and student organizations. Attended a meeting of a student club or organization.

16. 16 Self-assessed gains CCSEQ asks students to rate their progress on 23 key educational outcomes. Some of the outcomes are highly relevant to Student Service programs and activities: Gaining information about career opportunities Developing clearer career goals Understanding other people and the ability to get along with different kinds of people Developing good health habits and physical fitness

17. 17 Selected “activity” items from the CCSEQ (Community College Student Experiences Questionnaire) relevant to the diversity example. While in attendance at this college DURING THE CURRENT SCHOOL YEAR, about how often have you had serious discussions with students… who were much older or much younger than you whose ethnic or cultural background was different from yours whose philosophy of life or personal values were very different from yours whose political opinions were very different than yours Whose religious beliefs were very different than yours From a country different than yours. __ Never __ Occasionally __ Often __ Very Often The CCSEQ will provide benchmarks for comparison. Also the test publisher can provide comparison data from institutions that match your college’s student demographic makeup.The CCSEQ will provide benchmarks for comparison. Also the test publisher can provide comparison data from institutions that match your college’s student demographic makeup.

18. 18 Self-assessed gains CCSEQ asks students to rate their progress on 23 key educational outcomes. “Gain” items relevant to our diversity example include: Understanding other people and the ability to get along with different kinds of people.*** Becoming aware of different philosophies, cultures, and ways of life.*** Learning more about other parts of the world and other people (Asia, Africa, South American, etc.). Developing the ability to speak and understand anther language. There are creative ways to increase positive student to student contact. ESL students meet with students in Spanish classes. ***In a study done at SBCC, “The greater the breadth of contacts students had with students who differed from them in terms of age, ethnicity, philosophy of life, political and religious beliefs, and nationality, the greater the progress they reported…” on these (***) outcomes (Friedlander & MacDougall). In a commuter campus, many students, especially those who attend part-time or in the evening, don’t have much opportunity to interact with other students outside of class. There are a number of steps that colleges can take to increase interaction between students. Friedlander wrote about how students in ESL classes were paired with those in Spanish classes that met during the same time period. In the first half of the class period, the students in the Spanish class were asked to invite, in Spanish, the students in the ESL class to a movie. The vocabulary and grammar needed to accomplish this task required students to integrate and apply what they had been studying in class. Half-way through the class period, the students revered roles and the ESL students were asked to use English to invite the Americans to the movies. Activities such as these resulted in students in these classes extending their learning of the language outside of class through self-initiated activities like cross-tutoring, joint parties in which they shared cultural traditions, and attending college activities together. According to the students, these activities greatly increased their interest in and learning of the language. These activities also increased the probability that students would actually use the language they were learning outside of class and they would have meaningful interactions with students who backgrounds were very different from their own.***In a study done at SBCC, “The greater the breadth of contacts students had with students who differed from them in terms of age, ethnicity, philosophy of life, political and religious beliefs, and nationality, the greater the progress they reported…” on these (***) outcomes (Friedlander & MacDougall). In a commuter campus, many students, especially those who attend part-time or in the evening, don’t have much opportunity to interact with other students outside of class. There are a number of steps that colleges can take to increase interaction between students. Friedlander wrote about how students in ESL classes were paired with those in Spanish classes that met during the same time period. In the first half of the class period, the students in the Spanish class were asked to invite, in Spanish, the students in the ESL class to a movie. The vocabulary and grammar needed to accomplish this task required students to integrate and apply what they had been studying in class. Half-way through the class period, the students revered roles and the ESL students were asked to use English to invite the Americans to the movies. Activities such as these resulted in students in these classes extending their learning of the language outside of class through self-initiated activities like cross-tutoring, joint parties in which they shared cultural traditions, and attending college activities together. According to the students, these activities greatly increased their interest in and learning of the language. These activities also increased the probability that students would actually use the language they were learning outside of class and they would have meaningful interactions with students who backgrounds were very different from their own.

19. 19 Some CCSEQ Findings “the more effort students put into their college experiences, the more likely they are to make personal and academic gains” and “what students get of our college will depend, to a considerable degree, on the extent to which they take advantage of opportunities that are provided in the college setting” (Friedlander and MacDougall). We should encourage students to take advantage of enriching events offered by the college. We (administrators, faculty, support staff) should be role models by attending these events. Faculty should make sure we are offering quality opportunities/assignments in our classes. Quality of effort includes in-class and out-of-class experiences. For example, there was a correlation of 0.51 between the Writing Activities Effort Scale and the Communications Gain Scores. Many other correlations were even higher. BUT…OUR LAST CCSEQ STUDY (1997) – DURING THE CURRENT SCHOOL YEAR: 50% of the students in the sample had not used the library in the past year. 38% had never talked to one of their instructors outside of class 23% had never spent at least 5 hours writing a paper Point Three: The greater the use of the library as a resource and research tool, the greater the progress students reported toward developing the ability to learn on their own, pursue ideas, and find information they needed (independent inquiry skills) The greater the time and effort students devoted to writing, the greater the progress they reported making in learning how to write clearly and effectively. Astin said the CCs do a great job accommodating students, but sometimes maybe they bend back too far. Students deserve rigorous courses. Now we have baseline data and we have norms from colleges that are similar to ours in terms of demographics, etc.For example, there was a correlation of 0.51 between the Writing Activities Effort Scale and the Communications Gain Scores. Many other correlations were even higher. BUT…OUR LAST CCSEQ STUDY (1997) – DURING THE CURRENT SCHOOL YEAR: 50% of the students in the sample had not used the library in the past year. 38% had never talked to one of their instructors outside of class 23% had never spent at least 5 hours writing a paper Point Three: The greater the use of the library as a resource and research tool, the greater the progress students reported toward developing the ability to learn on their own, pursue ideas, and find information they needed (independent inquiry skills) The greater the time and effort students devoted to writing, the greater the progress they reported making in learning how to write clearly and effectively. Astin said the CCs do a great job accommodating students, but sometimes maybe they bend back too far. Students deserve rigorous courses. Now we have baseline data and we have norms from colleges that are similar to ours in terms of demographics, etc.

20. 20 CCSEQ

21. 21 LASSI Learning and Study Strategies Inventory

22. 22 LASSI Learning and Study Strategies Inventory Attitude – General attitude and motivation for succeeding in school and performing the tasks related to school success. Motivation – The degree to which one accepts responsibility for doing the specific tasks necessary for school success. Time Management – This scale measures the degree to which one creates and uses schedules. Anxiety – This scale measures how tense or anxious one is when approaching academic tasks. Concentration – Ability to concentrate and direct one’s attention to school and school-related tasks, including studying. Information Processing – Ability to create imaginal and verbal elaborations and organizations to foster understanding and recall. Selecting Main Ideas – Skill at selecting important information from the unimportant information that does not need to be remembered. Study Aids/Strategies - Ability to use or create study aids that support and increase learning and retention. Self-Testing –Testing one’s understanding. Test Strategies – Use of effective strategies to prepare for a test and other effective strategies useful while taking a test.

23. 23

24. 24 LASSI

25. 25 Student Exposure to Learning & Study Skills InstructionRudmann (1997) (N = 634 students)This shows that many students have had little, if any, instruction on strategies needed to succeed in college. CCSEQ – Students in the sample who reported have NO instruction (green bar) and SOME instruction (blue bar).CCSEQ – Students in the sample who reported have NO instruction (green bar) and SOME instruction (blue bar).

26. 26 Other data-gathering strategies Surveys Scannable approaches Online surveys Automatic telephone surveys Focus Groups

27. 27 Criteria for selecting assessment instruments

28. 28 Important criteria for selecting or creating a quantitative instrument (test) and extremely valuable resource documents Student service staff experts should do a content review of the items to assure they appear to measure the outcomes of interest. (Test names can mislead.) Test manual containing information on the test’s reliability validity norming studies test bias review Scoring efficiency Cost References Matriculation: Local Research Options - Written in 1991 and 1992, this series of three manuals remains an invaluable tool for assessment test validation and evaluation http://www.rpgroup.org/ Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges Available from the CCC Chancellor’s Office (Student Services unit) at: 28http://www.cccco.edu/divisions/ss/matriculation/attachments/stdpoprevalass.pdf You may have several experts on campus in this area: same criteria apply to matriculation placement tests. Matriculation researcher, IR person, certain English, math, ESL faculty may be quite experienced in this area.You may have several experts on campus in this area: same criteria apply to matriculation placement tests. Matriculation researcher, IR person, certain English, math, ESL faculty may be quite experienced in this area.

29. 29 Assessment considerations

30. 30 Categories of assessment Direct versus indirect assessment of learning outcomes Direct assessment of learning Usually means assessment using quantitative assessment tools – e.g., tests, rubrics. Indirect assessment of learning Some feel all qualitative assessments (e.g., surveys, focus groups, exit interviews) are indirect measures of learning. Assessment of achievement Graduation rates, transfer rates, course success rates, etc.

31. 31 Quantitative and Qualitative Assessment Options Quantitative and qualitative Quantitative examples… Scores on standardized or homegrown tests Scores derived from using a rubric to assess performance of some ability (e.g., a speech) Scores derived from an observation checklist Qualitative examples… Narrative drawn from a focus group Narrative and trends summarized from exit interviews

32. 32 Considerations for qualitative approaches to assessment Equipment People power Training

33. 33 Decisions to make, factors to consider regarding your assessment plan Voluntary or mandatory participation of students? Cross-sectional and/or longitudinal data gathering? Small or large random samples? Convenience samples? (Be careful here.) If you can track students, you can then compare outcomes for students who used services to a matching group of students who did not use the service. You’ll have a name bank of students to tap for surveys, focus groups, interviews, etc. Comparisons can include students who attended various multicultural events and those who did not; comparison of students who completed the college success/study skills course versus a match of students who did not; and so on.If you can track students, you can then compare outcomes for students who used services to a matching group of students who did not use the service. You’ll have a name bank of students to tap for surveys, focus groups, interviews, etc. Comparisons can include students who attended various multicultural events and those who did not; comparison of students who completed the college success/study skills course versus a match of students who did not; and so on.

34. 34 Triangulation Triangulation refers to the use of more than one measure for assessing progress on a given learning outcome. Triangulation is necessary because relying on a single measure (e.g., survey, interview, or standardized test) will always miss important aspects of what you are trying to measure.

35. 35 Example…using triangulation to assess why students drop out of college Interview students when they submit withdrawal forms at A&R. Send a questionnaire to parents. Give dropout students a personality or attitude scale. Interview teachers of advisors of dropout students. Compare institutional records of dropout students with those of persisting students.

36. 36 Wrong assumptions about assessment Whoever made the survey knew exactly what was important to ask the students. Everything important can be obtained through a single survey, standardized instrument, or interview. Students who participate in the data collection are representative of all students on campus. The information sought isn’t available elsewhere.

37. 37 Analyze and report the data The 1 page report (Jon Kangas) Title: outcome(s) assessed Method: brief summary of how and when data were gathered Results: Narrative, followed by table, followed by a graph Conclusion(s)

38. 38 It may take up to 3 semesters to: Identify the outcomes to assess Select instruments Create in-house instruments Plan out and pilot test any strategies using qualitative methods Create an efficient student tracking system Develop and pilot-test a documentation system Think of outcomes assessment as an on-going, term-by-term activity, not something to worry about every 6 years for the self-study.

39. 39 After one assessment cycle, evaluate the methods used and the results obtained Are your assessment tools appropriate or do they need modification? Efficient, cost-effective? Regarding the learning outcomes: retain, modify, delete, add? Are your sampling strategies appropriate or need adjustment? It may take several cycles to perfect the entire process.

40. 40 Some sources of learning outcomes in Student Services

41. 41 SLO Sources in Student Services: Clubs Biology – Weekend field trips Criminal Justice – National competitions Geology – Weekend field trips Political Science – Model United Nations Psychology – Attend professional conferences Miscellaneous clubs – Ethnic groups, hobbyists, recreational, MTE (Making Transfer Easy)

42. 42 It is important to acknowledge that Student Service units often play a key role by supporting and enabling many educationally enriching opportunities for our students. These events and activities often are rich sources of learning.

43. 43 At many 2-year colleges Psi Beta (Psychology) students attend professional conferences and present their behavioral research studies. Students receive recognition from the National Psi Beta Office The student research feedback poster session was developed by Lisa Grey Shellberg and was launched at the WPA in LA (1995?). It involved a lot more undergraduates than Psi Beta and Psi Chi honor students and was an excellent experience. Starting next year in Hawaii, let’s make these a routine feature of the WPA annual meeting.The student research feedback poster session was developed by Lisa Grey Shellberg and was launched at the WPA in LA (1995?). It involved a lot more undergraduates than Psi Beta and Psi Chi honor students and was an excellent experience. Starting next year in Hawaii, let’s make these a routine feature of the WPA annual meeting.

44. 44 SLO Sources in Student Services: Matriculation Orientations Students receive information about… Campus Resources Adding and dropping classes Transfer information Interpretation of assessment results Etc.

45. 45 SLO Sources in Student Services: Peer Tutoring Tutors must complete a course on tutoring How to help peers prepare for tests, manage time, etc. Tutors themselves acquire deeper understanding of material

46. 46 SLO Sources in Student Services: Serving on College Governance Committees Examples: Academic Senate Curriculum Committee DSP&S Advisory Committee Research Advisory Committee

47. 47 SLO Sources in Student Services: Interactions with Staff Student affairs professionals often serve as role models. They also… explain procedures, serve as advocates, provide advice and assistance.

48. 48 SLO Sources in Student Services: Student Government Some colleges offer leadership classes that are transferable as electives to CSU. Content: Improve leadership and management skills (Students earn a leadership certificate & 12 units) Students attend leadership conferences and serve on college governance committees SLO Resource: Miller, T.K. (ed), (1997). The Book of Professional Standards for Higher Education. Washington, DC: Council for the Advancement of Standards in Higher Education. [www.cas.edu] This resource provides a list of student leadership SLOs organized under 1) foundations of leadership, 2) personal development, and 3) organizational development.

49. 49 SLO Sources in Student Services: Workshops Health Office presentations… Eating disorder screening and workshop Depression screening workshop Safe sex, sexually transmitted diseases Stress management College Transfer Workshops

50. 50 SLO Sources in Student Services: Work Study Students work in various jobs at the college (funded by EOP&S and Financial Aid Office). Some of these may be sources of learning outcomes for students.

51. 51 Hypothetical outcomes, educational strategies, and and assessment plan for campus diversity Understanding and Appreciation of Diversity

52. 52 Students will exhibit tolerance and acceptance of persons from different cultural, racial, and religious heritages.  Students will be able to… Evaluate their current levels of acceptance for culturally different people. Exhibit knowledge of diverse human cultures. Interact with people from different cultural heritages. Accept feedback from different cultural heritages. Provide feedback to others from different cultural heritages. Identify and distinguish between examples of artistic expression from diverse cultures. Describe different religious beliefs throughout the world. Evaluate currently held biases, prejudices, and stereotypical thought.

53. 53 Hypothetical strategies for developing an understanding and appreciation of diversity on the campus. All GE courses include at least one tolerance development lesson. Students encouraged to participate in multi-cultural programs and activities. Student orientation includes a section on institutional expectations concerning relations among students from differing ethnic backgrounds. Leaders of tolerance development activities purposely includes staff and students from differing cultural and racial backgrounds. Community service activities are designed to bring together students of differing backgrounds.

54. 54 A sample plan assessment of tolerance for racial, ethnic diversity This is a hypothetical plan for assessing the development of tolerance on the campus. Note how this plan reflects method triangulation – we have 4 different sources of assessment information on the outcome of diversity tolerance.This is a hypothetical plan for assessing the development of tolerance on the campus. Note how this plan reflects method triangulation – we have 4 different sources of assessment information on the outcome of diversity tolerance.