Assessment in Student Affairs: Some Thoughts on a Tricky Business

1. Assessment in Student Affairs:Some Thoughts on a Tricky Business Peter T. Ewell National Center for Higher Education Management Systems (NCHEMS) Student Affairs Assessment Conference Emory University October 14, 2013

2. Logic of This Keynote Challenges and Dilemmas of Assessment Practice in Student Affairs: What Makes This a Tricky Business Growing Demand for the Kinds of Proficiencies at the Center of Student Affairs Some (Occasionally Odd) Techniques for Getting at These Proficiencies Using Results of Assessment to Improve Programming and Practice in Student Affairs

3. A Prefatory Comment Two Worlds of Assessment: Academic: Faculty Want to Know About that Small Piece of the Student that is Trying to Learn [Mathematics, History, etc.] Student Affairs Professional: Want to Know About the Experience and Development of the Whole Student that is Experiencing an Emotional, Attitudinal, and Cognitive Process of Transformation

4. Some Particular Challenges of Assessment in Student Affairs • The Domains of Interest are Not Easy to “Measure” • Many of Them are Affected by Dispositions to Act in Particular Ways Even Though Students “Know” What they Should Do • Attributions of Cause are Hard to Relate to Individual or Isolable Programmatic Features or Elements

5. The Growing Demand for These Kinds of Proficiencies Employer Complaints About Recent College Graduates [and What They Want Instead] Demands for Integrated Abilities and Practical Competence Reflected in Prominent Expectations Frameworks Like the Lumina Degree Qualifications Profile (DQP) and AAC&U’s LEAP Goals

6. Graduate Attributes: Top Choices for Employers from 2013 AAC&U Survey Percent Reporting “Very” and “Somewhat” Important: Ethical Judgment and Integrity – 96% Comfortable with Diverse Backgrounds – 98% Demonstrated Capacity for Professional Development – 94% Interest in Giving Back to Communities Our Company Serves – 71%

7. Some Proposed Revisions to DQP 2.0 Confronts the ethical issues present in prominent problems in politics, economics, society, scholarship, or the arts and demonstrates how at least two different ethical perspectives, influence decision making with respect to such problems Describes how knowledge from different cultural perspectives would affect inter­pretations of prominent problems in politics, society, the arts and/or global relations

8. Some AAC&U LEAP Goals Ethical Reasoning and Action Inter-Cultural Knowledge and Competence Teamwork and Problem Solving Civic Knowledge and Engagement – Global and Local Foundations and Skills for Lifelong Learning

9. Some Useful Approaches for Gathering Evidence About Values and Dispositions Survey Items and Item Construction Time Diaries and the “Beeper Study” Formal Focus Groups and Interviews Informal Observation and Imagery Checklists Rubrics Applied to Behavior and Work Products Inventories of Good Practice

10. Survey Items and Item Construction Scales with Even Numbers of Decision Points “Semantic Differential” Items Scenario Based Items (e.g. The Defining Issues Test) Open-Ended Items (e.g. “Areas and Agents of Change”)

11. Semantic Differential Items Example

12. “Areas and Agents of Change” Example

13. Observation and Imagery Develop and Pilot Clear Observational Protocols Ensure What You are Looking at is Typical For Ongoing Processes, Observe Regularly at Defined Points in Time Consider Using Photographs or Videos to Capture Demographic Mixes or Ongoing Patterns of Behavior

14. Observation and Imagery Develop and Pilot Clear Observational Protocols Ensure What You are Looking at is Typical For Ongoing Processes, Observe Regularly at Defined Points in Time Consider Using Photographs or Videos to Capture Demographic Mixes or Ongoing Patterns of Behavior

15. Some Applicationsof Checklists • Academic Advising • Service Learning Courses • Internships, Field Placements • Student inquiries • Academic Skills Centers • Residence Hall and Student Group Advisors

16. AAC&U VALUE Rubric for Teamwork

19. Evidence-Based Management: What Does a “Culture of Evidence” Require? • Respect for the Facts • Clear Goals and Objectives • Concrete Evidence About the Achievement of Goals and Objectives • An Attitude Toward Problem-Solving that Avoids “Finger-Pointing” • Clear Follow-Through On Decisions and Why They Were Made • Willingness to Stop Doing Things When They Don’t Work

20. Strategies for Addressing Key Difficulties in Building a Culture of Evidence • Keep the Outcomes Framework Simple • Look at Data Distributions, Not Just “Averages” • Adjust the Approach to Accommodate Important Differences in Context • Share Approaches Across Units • Don’t Wait for Perfection

21. Some “Points of Attack” in Building Cultures of Evidence • Expectations Exercises • Use Results to Inform and Initiate Discussions, Rather than “Give Answers” • Initiate Visible Opportunities to Consider Results and What They Mean • Present Results as Discrepancies • Disaggregate Results to Show Variations • Package Results Around Perceived Problems or Embed Them in Regular Decision Processes

22. Some Final Reminders If It’s Important, Try to Assess It – No Matter How “Tricky” the Business Seems to Be Dispositions Matter – So Experiment with Indirect and Unobtrusive Approaches Use Multiple Forms of Evidence Wherever Possible Be Creative and Have Fun