Assessment Institute, Summer 2011

1. Assessment Institute, Summer 2011 Heather McGovern

2. Overview • Defining “assessment of student learning” • Writing goals • Measuring learning • Applying information

3. What is assessment of student learning?

4. Assessment of student learning is “A natural, scholarly act that can bring important benefits” with 3 steps (Walvoord 2010) or 4 steps (Suskie): Where is your project in terms of these steps? Teaching

5. What common changes occur because of assessment? • “Changes to curriculum, requirements, programmatic structures, or other aspects of the students’ course of study.” Examples!—WGSS, POLS, PHIL, COMM, MATH • “Changes to the policies, funding, and planning that support learning” Examples—class size, qualifications of teachers, tutors • “Faculty development” Examples! Writing (information literacy) (Walvoord) See list of questions assessment can inform in Suskie, pgs. 65-66 What changes have you/your program made because of informal or formal assessment?

6. What is bad assessment? Write a minute paper on your fears about and bad experiences with assessment. “Mere compliance with external demands” “Gathering data no one will use” “Making the process too complicated” (Walvoord)

7. What does a good assessment do? Write a minute paper about your hopes for and good experiences with assessment • “yield reasonably accurate and truthful information about what students have learned, so that we can use the assesment results with confidence to make plans and decisions” • “have a clear purpose, so that [they] are valued and don’t end up sitting on a shelf • “Engage faculty and staff, so the assessment becomes a useful part of the fabric of campus life” • “Flow from and focus on clear and important student learning goals, so the results provide information on matters the college or university cares about” (Suskie, pg. 37)

8. Writing Goals

9. Writing goals: What do we want students to know or be able to do? I We might consider goals at the institutional level, school level, program level, or course level (Walvoord pg. 14) We can identify learning goals through our own reflection or collaboration with other faculty, staff, and/or students to reach consensus (pg. 127, Suskie—see discussion questions on pg. 128)

10. Writing goals: What do we want students to know or be able to do? II Which of the items on this slide are you most interested in? Reflect for a moment on your own, then we’ll share. We might think about goals in terms of Kirkpatrick’s 4 levels of learning experience outcomes (in Suskie, pg. 22): • Reaction • Learning • Transfer • Results We might consider formative items like a) learning processes (pg. 25), b) inputs (pg. 26), and C) context (pg. 27) as well as the summative d) outcomes (“knowledge (knowledge, skills, attitudes, and habits of mind”) (Suskie, pg. 23) Note, I recommend you don’t get caught up in vocabulary, but if you want to, you can read Suskie pg. 116-117

11. Which of the items on this slide are you most interested in for your project? Write down all the terms you think might apply. Common learning outcomes (Suskie) • Knowledge and conceptual understanding (pg. 118): “remembering, replicating a simple procedure, and defining, summarizing, and explaining concepts or phenomena” • Thinking and other skills (pg. 120): application, analysis, evaluation, problem-solving, decision-making, synthesis, creativity, critical thinking, information literacy, performance skill, interpersonal skills • Attitudes, values, dispositions, and habits of mind (pg. 123): “appreciation, integrity, character, enjoying and valuing learning, becoming more aware of one’s values, attitudes, and opinions and their evolution,” metacognition, “capacities to work independently, set personal goals, persevere, organize, be clear and accurate, visualize, be curious”

12. Writing goals: language I Walvoord, pg. 14: Say “Students will be able to….” NOT The curriculum emphasizes… The institution/program/course values Students are exposed to… Students participate in…

13. Writing goals: language, II With a partner, write 2 good learning goals. More advice in Suskie, pgs. 129-132 Bottom line—try using action-word goals where possible. Try not to be too broad or too specific. Sometimes, broader words like “know,” “understand,” “appreciate” are appropriate when combined with a rubric or rating scale. For example, trying to break “communicate effectively in writing” down might result in so many specifics that the major goal is lost. (Suskie, pg. 131)

14. Measuring learning

15. Two main kinds of data • Indirect (surveys (NSSE and others), grades, retention/graduation rates, placement rates, IDEA forms, GRE scores, etc.) • Direct (field or employer ratings; scores and pass rates on licensure exams or Major Field Tests; capstone experiences, portfolios, written work, performances, projects, etc, scored using a rubric; score gains on local tests or the CLA; systematic observations of student behavior) See table in Suskie, pg. 21

16. Connect kind of data to measurement (Suskie, pg. 34) What kinds of measurement might be appropriate for each of the two learning goals you have? Why or why not? Discuss with your partner. • Attitudes/values/ disposition/reaction • Thinking/Performance skills • Knowledge/conceptual understanding • analysis/application • Survey, focus group, interview, reflection • Assignments with rubric, performance task • Multiple choice tests • Assignments with rubric, performance task or case study, portfolio

17. Toolbox for measuring • Scoring guide/rubric for an assignment, performance, portfolio, performance task, etc. (Chapter 9, Suskie) • Assignment for assessment (Chapter 10, Suskie) • Making a local test for assessment—particularly see the test blueprint on pg. 167 (Chapter 11, Suskie) • Portfolio for assessment (Chapter 13, Suskie) • Using a published test or survey (Chapter 14, Suskie) • Using reflective writing, short questions, prompts, journals, interviews, focus groups, surveys (Chapter 12, Suskie)

18. Two (or three) ways to get your data • Use existing student work/tests/surveys/IDEA/NSSE/CLA • Embed new assignments/tests/surveys in classes • Add-on work

19. Program default plan, Walvoord • Have learning goals for each degree • Have two measures of how students are meeting each goal, one direct (Walvoord suggests a sample of student work at the end of the course of study) and one indirect (Walvoord suggests a student survey or focus group) Read chapter 3, Walvoord. See the list of examples, pg. 64-66

20. Side note: General studies considerations, Walvoord -Each unit makes course learning goals, assignments and tests, and ways to measure these consistent with the general education goals -Each unit regularly sends a two page report on strengths/concerns/actions to a body or committee -Institution-wide survey like NSSE Suskie notes things that make assessing general studies difficult, pgs. 110-111.