1 / 36

University of Florida Accreditation Process

Developing a QEP Assessment Plan: Criteria and Process of Instrument Review and Development M. David Miller, Director, QEP Timothy S. Brophy , Director, Institutional Assessment University of Florida. University of Florida Accreditation Process. Core Requirement 2.12.

miriam
Download Presentation

University of Florida Accreditation Process

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developing a QEP Assessment Plan: Criteria and Process of Instrument Review and DevelopmentM. David Miller, Director, QEPTimothy S. Brophy, Director, Institutional AssessmentUniversity of Florida

  2. University of Florida Accreditation Process

  3. Core Requirement 2.12 The institution has developed an acceptable Quality Enhancement Plan that includes an institutional process for identifying key issues emerging from institutional assessment and focuses on learning outcomes and/or the environment supporting student learning and accomplishing the mission of the institution. (Quality Enhancement Plan)

  4. Standard 3.3.2 The Institution has developed a Quality Enhancement Plan that (1) demonstrates institutional capability for the initiation, implementation and completion of the QEP; (2) includes broad-based involvement of institutional constituencies in the development and proposed implementation of the QEP; and (3) identifies goals and a plan to assess their achievement.

  5. Begin with the end in mind QEP Assessment

  6. Procedure for Planning Assessment

  7. Types of Assessments

  8. Our Guiding Principles

  9. Our Guiding Principles, Part II

  10. Step 1: Define Theme and SLOs • UF Theme : Internationalization Internationalization is the conscious integration of global awareness and intercultural competence into student learning.

  11. Step 1: Define Theme and SLOs

  12. Step 2: Identify Existing Instruments • Internationalization Measures Identified – Commercially Available • Global Perspectives Inventory (GPI) • Global Competence Aptitude Assessment (GCAA) • Intercultural Development Inventory (IDI) • Global Competencies Inventory • Cross Cultural Adaptability Inventory • Global Awareness Profile • Intercultural Effectiveness Scale

  13. Step 2a: Identify Experts for Review • International Task Force • Representation from all 16 Colleges • Student Representation • Representation from Faculty, Staff, Students, Administration • Assessment Committee (5-6 with expertise on content and assessment)

  14. Step 2b: Defining Review Criteria

  15. Step 2c: Factors in our decision to create our own assessments

  16. Existing Surveys on Campus • Consider Options Already Being Used as Part of Reporting System • Student Experience in the Research University (SERU) - biennial survey used by multiple universities that includes items on internationalization

  17. SERU • Items measure behaviors (e.g., courses taken, participation in Study Abroad, and other types of international experiences) and attitudes toward other cultures • We elected to add 10 items designed to measureattitudes related to SLOs 2 and 3

  18. What Assessments do we need to Develop?

  19. Developing the Assessments

  20. Step 3a: Indirect Assessments

  21. Indirect Assessment Development Steps 1.Development of item specifications based on the two SLOs and a literature review. 2.Writing items based on the item specifications. • Approximately 70 items were written for each SLO

  22. Indirect Assessment Development Steps 3.Review of the items by the ITF, the Assessment Committee and other experts in assessment. (Validity and Match to SLOs) 4. Revision of items based on feedback from expert review. • Revisions were minor changes in wording.

  23. Indirect Assessment Development Steps 5. Pilot testing with undergraduate students at UF and eliminating items with poor discriminations. • Initial piloting was completed with four forms to minimize the testing burden for students. • Forms overlapped with ten items that expert agreed helped to define the construct. • Each form was pilot tested with 70-100 undergraduates. 6. Item analysis of pilot data. • The scale reliabilities exceeded .95 for all four forms. • Items were retained that had an item discrimination of .25 or higher.

  24. Indirect Assessment Development Steps 7. Pilot testing (N=70-80) the retained items on a single form for each SLO. 8. Item analysis of pilot data. • Recommended retaining items with the highest item discriminations that would result in a scale with a reliability of at least .90. • For IntCRIT, the recommendation was to retain 12 items. • For IntCOMM, the recommendation was to retain 14 items. 9. Review of the items by the ITF, the Assessment Committee and other experts in assessment. (Validity and Match to SLOs)

  25. IntCRIT • I consider different perspectives before making conclusions about the world. • I am able to manage when faced with multiple cultural perspectives. • I am open to different cultural ways of thinking in any international context. • I can make effective decisions when placed in different cultural situations • Knowing about other cultural norms and beliefs is important to me. • I am able to think critically to interpret global and intercultural issues. • I actively learn about different cultural norms. • Understanding different points of view is a priority to me. • I can recognize how different cultures solve problems. • I can contrast important aspects of different cultures with my own. • Knowing about other cultural beliefs is important. • I am able to recognize how members of other cultures make decisions

  26. IntCOMM • I demonstrate flexibility when interacting with members of another culture. • I prefer to socialize with people of my culture. • I am confident that I can adapt to different cultural environments • I am able to communicate effectively with members of other cultures • I like working in groups with students from other countries. • I feel comfortable in conversations that may involve cultural differences. • When working on a group project, I enjoy collaborating with students from other countries. • I often ask questions about culture to members of other cultures. • I enjoy learning about other cultures • I appreciate members of others cultures teaching me about their culture. • I am able to interact effectively with members of other cultures. • I appreciate differences between cultures • I feel comfortable discussing international issues. • I can clearly articulate my point of view to members of other cultures

  27. Step 3b: Direct Assessments

  28. Adopt General Education model on campus

  29. Developing Rubrics • The Association of American Colleges and Universities developed 15 Rubrics that can be used across programs and courses • VALUE (Valid Assessment of Learning in Undergraduate Education) rubrics were developed by faculty and assessment expert teams across the country -- used by more than 2000 institutions (http://www.aacu.org/value/index.cfm)

  30. Adapting VALUE Rubrics for SLOs

  31. Additional Steps

  32. Outputs • Number of participants at specific campus events with an international focus (QEP events). • Number of International Scholar courses. • Number of students enrolled in International Scholar courses. • Number of Study Abroad courses offered. • Number of students studying abroad.

  33. Assessment Plan

  34. Closing the Loop

  35. For Information Contact: M. David Miller dmiller@coe.ufl.edu Timothy S. Brophy assessment@aa.ufl.edu

More Related