SUNY GENERAL EDUCATION ASSESSMENT CONFERENCE

1. SUNY GENERAL EDUCATION ASSESSMENT CONFERENCE Guidelines for and Implementation of Strengthened Campus-Based Assessment

2. Presenters: Patricia Francis, Melanie Vainder, and Tina Good GEAR Co-Chairs

3. Session Objectives: • To enable participants to return to their institution with a clear idea of how to begin the process of revising their campus’ existing assessment plan to meet the new GEAR guidelines • To begin dialogue among ourselves – focusing on best assessment practices – as we move toward implementing Strengthened Campus-Based Assessment

4. Specific Topics to be Covered: • Clarification of “how the process will work” with special emphasis on issues of concern raised by campuses • Using nationally-normed measures and correlating a local measure to a nationally-normed measure: Issues to consider and advantages/disadvantages • Using scoring rubrics and standards: Issues to consider and advantages/disadvantages

5. How the Process Will Work: The New Guidelines Patricia Francis, Assistant Provost for University Assessment and Academic Initiatives

6. Strengthened Campus-Based Assessment: Major Implications • One general education assessment process, overseen by GEAR • Utilization of externally referenced measures for Basic Communication [Written], Critical Thinking [Reasoning], and Mathematics, effective Fall 2006 • Measure of campus’ academic environment • Option of using value-added approach • Cost to be covered by System Administration (with sample size limitations consistent with existing GEAR guidelines)

7. GEAR’s #1 Operating Principle: Require as few changes as possible in campus’ existing general education assessment plan (and, therefore, minimal new information)

8. Campus Responses to Draft GEAR Guidelines Concerns and Answers

9. Funding • System Administration will bear the cost of all three measurement options, based upon a sample size equal to at least 20% of total students enrolled in a learning outcomes area at the time of the assessment • System Administration will also fund the administration of the NSSE, CCSSE, or other measure of academic environment

10. Mathematics Learning Outcomes • For Strengthened Campus-Based Assessment, campuses will develop plans that focus on the new math outcomes approved by ACGE and the Provost • These outcomes can be found in your registration packet

11. Mapping of Existing Nationally-Normed Measures to SUNY Learning Outcomes • GEAR concluded there was inadequate mapping during Fall 2004 • In meetings between System Administration staff and testing company representatives, we emphasized the importance of adapting measures to meet SUNY’s needs

12. Course-Embedded Assessment as an Assessment Strategy • GEAR has always encouraged campuses to use course-embedded assessment, and will continue to do so (though campuses are certainly free to propose and use alternative approaches)

13. Integrating New Campus Plans Into Existing Campus-Based Plans • Campuses already have GEAR-approved plans, and much of what is included in those plans need not be changed • In particular, campuses should feel free to adhere to their existing assessment schedule • The major change: Effective Fall 2006, campuses must use externally referenced measures as approved by GEAR to assess Writing, Critical Thinking, and Mathematics

14. Options 1 and 2: Using Nationally-Normed Measures and Correlating a Local Measure to a Nationally-Normed Measure Melanie Vainder, Professor of English and Technical Communications, Farmingdale State University

15. ACT CAAP ACADEMIC PROFILE CALIFORNIA CRITICAL THINKING SKILLS TEST QUANT Q CRITICAL REASONING APPRAISAL GRE ACCUPLACER (Including WritePlacer) GEAR Research: Existing Nationally- Normed Measures

16. Using Nationally-Normed Measures: Advantages • Less labor intensive with respect to test development and scoring (particularly in the area of writing), and reliability of scoring assured • Provides opportunity for campuses to compare results with those obtained at peer institutions   • Reporting capacity provided by companies, allowing campuses to examine overall program effectiveness, success of individual courses, and relationship between student variables and performance

17. Using Nationally-Normed Measures: Advantages (cont.) • Relative ease of using pre- and post-test approach in order to determine “value added” if desired   • Ability for campuses to choose from among available modules in the areas of Writing, Mathematics, and Critical Thinking (i.e., it is not an “all or nothing” approach)   • Possibility of using measures in a course-embedded fashion, completed within a single class session

18. Using Nationally-Normed Measures: Disadvantages • Problems with student motivation in stand-alone testing • Existing measures do not map adequately to the SUNY Learning Outcomes for Writing, Mathematics, and Critical Thinking • Existing measures do not yield separate sub-scores for each of the Learning Outcomes for Writing, Mathematics, and Critical Thinking

19. Correlating a Local Measure to a Nationally- Normed Measure: Issues to Consider • Does the local measure directly assess student learning and does it measure the learning outcome(s) it is intended to measure? • Is it characterized by adequate inter-observer reliability?  • Has it been demonstrated to correlate statistically with a nationally-normed measure of the same learning outcome(s)?

20. Correlating a Local Measure to a Nationally- Normed Measure: Advantages • Closer alignment between locally-developed measures and curriculum • Local measure can be specifically developed to meet all SUNY Learning Outcomes • Possibility that campuses may continue to use previously-used measures (and therefore be able to make direct comparisons between student performance on the same measure)

21. Correlating a Local Measure to a Nationally- Normed Measure: Disadvantages • Duplicate testing will be needed at outset to demonstrate correlations between local and nationally-normed measures • Very time- and labor-intensive • Student motivation factor • Extensive psychometric expertise required with this approach

22. Using Scoring Rubrics and Standards Tina Good, Assistant Professor of English, Suffolk County Community College

23. Option 3: Using Scoring Rubrics and Standards • Discipline-Specific Panels are working to create rubrics and standards for: • Written Communication • Mathematics • Critical Thinking • Process of rubric design will be transparent • Drafts of rubrics will be posted online • Minutes and membership are posted online

24. Using Scoring Rubrics and Standards:Options • Use the actual rubrics and standards created by Discipline-Specific Panels • Show how your campus rubrics correlate to the rubrics designed by the panels • Mix and match

25. Using Scoring Rubrics and Standards:Advantages • Provides an opportunity to re-submit already developed rubrics and demonstrate correlations with those designed by panels • Provides for faculty involvement in the creation of rubrics and standards for their own programs • Allows for revision of rubrics as innovations, philosophies and pedagogies evolve in the discipline

26. Using Scoring Rubrics and Standards: Advantages • Provides for faculty involvement in the assessment process (i.e., through application of the rubrics) • Rubrics can be specifically developed to meet all SUNY Learning Outcomes • Provides for collaboration on multiple levels throughout the assessment process

27. Using Scoring Rubrics and Standards:Disadvantages • Assessment process more cumbersome to implement than a nationally-normed measure • The level of faculty involvement required could also be a disadvantage, especially for those programs that have few faculty available to serve on assessment committees • Establishing validity and reliability of process can be time consuming

28. Implementing Strengthened Campus-Based Assessment: Resources • The GEAR Group and Web site (www.cortland.edu/gear) • SUNY System’s Office of Academic Affairs • Sister campuses – many “best practices” are already out there! • Discipline-Specific Panels • Other ideas?

29. SUNY GENERAL EDUCATION ASSESSMENT CONFERENCE Guidelines for and Implementation of Strengthened Campus-Based Assessment