Download
1 / 36
360 likes | 488 Views
William Carey University Faculty Development. Staying ahead in an era of assessment Robert G. Cuzzolino , Ed.D . Vice President for Graduate Programs and Planning, PCOM. Osteopathic Medical Education in an environment of planning and assessment. Objectives.
E N D
William Carey UniversityFaculty Development Staying ahead in an era of assessment Robert G. Cuzzolino, Ed.D.Vice President for Graduate Programs and Planning, PCOM
Osteopathic Medical Education in an environment of planning and assessment
Objectives Be able to distinguish among mission, goals and outcomes Describe how a continuous quality improvement program works. Organize goals or competencies with related outcomes data in an understandable way Describe the conditions necessary for strategic planning to yield organizational or instructional change.
Objectives, continued Describe methods of longitudinal analysis Differentiate between the measurement of student achievement and the assessment of institutional or program effectiveness
Mission, Goals, Competencies, Outcomes • External reviewers focus on mission as the foundation for institutional assessment and planning. • Mission statements are just a starting point in the assessment/planning process chain.
Mission • Goals • Competencies • Student Outcomes • Program Assessment • Planning
Mission demands goals • Goals demand learning objectives • Objectives demand measurement • Measurement demands analysis • Analysis demands planning • Planning demands change • Change demands review in context of mission
Be kind to your outside reviewers. • The linkages among institutional goals, program competencies and outcomes are not intuitive to most visitors, so why not map them out?
On Change • 1.3. “The COM must connect its strategic plan to learning outcomes assessment in order to continuously improve the educational quality of its osteopathic medical education program.” • In other words, “How does my COM make changes?” “Have the changes we’ve made been based on outcomes data?”
Assessment Data / SWOT Analysis /Departmental Plans /Community Input / Committee Actions/ Accreditor Recommendations/ Budget Requests PlanningResource Allocation Responsibility for Change Progress Reporting Annual Plan Renewal
Outcomes, outcomes and more outcomes “Constant testing will no more address the problems with our educational system than constantly putting an overweight person on the scale will cure obesity.” -Anna Quindlen
Agreeing on the terminology • There is a difference between the measurement of individual student achievement and theassessment of institutional or program effectiveness. • Unfortunately, accreditation standards often refer to both of these as“learning assessment” or “outcomes.”
COCA Standard 6.6:“The COM must define, publish and implement educational outcomes, based on its own educational objectives that will prepare students for osteopathic graduate medical education.”
Your COM can not measure learning outcomes if it has not articulated its learning objectives.
Standard 6.5 Documentation guideline for proving that your COM has stated learning objectives and assessment methods:“Course objectives and course syllabi, evaluation process for all courses, and outcomes from the evaluation process.” • Standard 6.6.1“The COM must establish clinical competencies and a methodology to ensure they are being met.”
Learning Assessment Plans • Most institutions have or are developing learning assessment plans. • These plans tend to focus on the individual student level and define the expected competencies, assessment methods, timing and (sometimes) explain how these outcomes are reported back to the college.
Learning Assessment Plans are valuable at the individual student assessment level. • They document the measurement of “learning” up front for students, and also compel faculty to clearly define objectives and develop measures of student achievement - - as a system. • While they are a good basisfor developing objectives and corresponding measures of learning, assessment plans do not produce program effectiveness data. They illustrate how you evaluate students… not the program.
Question for the group What have you learned from Comlex I results?
200 Comlex scores mean nothing in and of themselves. But if the lowest 50 scores are your PBL track students, you have a program effectiveness issue
When does a set of learning assessment data help in planning? • When it is aggregated • When it is augmented by other data • When it illustrates a trend • When it is benchmarked or comparative • When it suggests a course of action, a change, or the need for a resource.
What learning outcomes data sources can help inform institutional change? • Comlex scores are a mandated element. • Others?
Student clerkship ratings • Course grade trends • Standardized patient encounters • Student survey data • End of clerkship exams • Alumni surveys • AOA, state board data • Freshman surveys & Graduation Surveys • Match data and residency selection data • Certification
Council for Higher Education Accreditation: • “Effectiveness is a broad concept that refers to the overall attainment of the mission and goals of a particular institution or program. As such, it may embrace various kinds of behavioral outcomes for students that go beyond student learning, such as employment, economic mobility, and contributions to personal and civic life…”
Summary on outcomes data • In the self study, we have two needs for academic performance data: First – Student Assessment WCUOM must show that individual student achievement is regularly measured against the expected competencies in a planned manner. It is also important that we indicate when and how these expectations and their assessments are communicated.
Second – Program Evaluation The self study must coalesce individual student achievement results into meaningful program assessment data sets, and then show that the COM analyzed the outcomes data and took the appropriate action. Link curricular, instructional, organizational, and resource changes to data.
Example of an institutional effectiveness process – COMLEX and DO student success This brief presentation will share information that was developed by the PCOM learning resource committee and shared with the DO Curriculum Committee and the DO Admissions Committee. It departs from analysis of grades, clinical observation ratings and mean Comlex scores and treats retrospective data to inferential analysis, with conclusions as to meaning, but leaves the action plan to the faculty.
Tracking Change This is most difficult aspect of continuous quality improvement through strategic planning.
Longitudinal data • What do you (will you) know about practice characteristics (geographic distribution, specialty, residency match, certifications) of your graduates ? • Think ahead – define your questions before you graduate a class.
Please rate the graduate, «FIRST_NAME» «LAST_NAME», on each of the following items by circling the appropriate number. In making the ratings please compare the intern with all interns you have supervised, not just those in your recent group: Top Upper Lower Bottom Unable to Quarter Middle Middle Quarter Judge Quarter Quarter Osteopathic Principles and Practice, which are established and incorporated in the development of skills. ____ ____ ____ ____ ____
Strategic Plan • Goal #2: Enhance academic quality, faculty development, scholarly activity and clinical service. k) Pursue state approval of interdisciplinary PhD research degree or pursue partnership with PhD-granting institution (see Goal 5 a) l) DO curriculum committee to assess the adequacy of business in medicine content and encourage co-curricular presentations on business, management and medical economics. m) Convert geriatric lecture content to on-line format; reformat course to focus on interactive discussion
Summary on assessment Demonstrating the connection of learning outcomes to strategic planning is important. Student assessment and program assessment are not the same thing, but both are important to the self study, and both run on their own respective form of learning outcomes data. The COCA standards carry an expectation of longitudinal data.
