1 / 68

Outcomes Assessment 2 Program Assessment

Outcomes Assessment 2 Program Assessment. Joseph A. Shaeiwitz West Virginia University joseph.shaeiwitz@mail.wvu.edu. Daina M. Briedis Michigan State University briedis@egr.msu.edu. Outline. ABET and engineering criteria Program objectives Program outcomes Assessment

akeem-hardin
Download Presentation

Outcomes Assessment 2 Program Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Outcomes Assessment 2Program Assessment Joseph A. Shaeiwitz West Virginia University joseph.shaeiwitz@mail.wvu.edu Daina M. Briedis Michigan State University briedis@egr.msu.edu

  2. Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review

  3. Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review

  4. Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.

  5. Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.

  6. Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.

  7. Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.

  8. Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.

  9. ABET and Engineering Criteria • ABET = Accreditation Board for Engineering and Technology • Engineering criteria changed to assessment basis (TQI) around 2000 • Must prove that students are achieving objectives and outcomes • measure output, feedback model • previously, feed forward model

  10. Feed Forward Model curriculum input output assumed

  11. Feedback Model education process one class one course graduate alumnus one class entering college one course

  12. Determine educational objectives Determine Outcomes Required to Achieve Objectives Determine How Outcomes will be Achieved Evaluate/Assess Determine How Outcomes will be Assessed Input from Constituencies Formal Instruction Student Activities Establish Indicators that Objectives are Being Achieved The Two Loops of the Engineering Criteria

  13. ABET and Engineering Criteria • 8 criteria • students • program educational objectives • program outcomes and assessment • professional component • faculty • facilities • institutional support and financial resources • program criteria

  14. ABET and Engineering Criteria • Focus of this workshop • program educational objectives • definition • how to establish • how to assess • program outcomes and assessment • definition • how to establish • how to assess

  15. Minute PaperClearest vs. Muddiest Point • What have you just learned about ABET and assessment? • What points are the clearest? • What points are the “muddiest?”

  16. Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review

  17. Program Objectives* • “…broad statements that describe the career and professional accomplishments that the program is preparing the graduates to achieve.”* • “Our graduates will be successful …” • must define “successful” *Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org

  18. Program Objectives* • must be • detailed and published • include constituencies/periodically evaluated • educational program to achieve outcomes (defined later) and to prepare graduates for accomplishments that achieve objectives • ongoing evaluation to determine extent objectives attained, use results for program improvement *Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org

  19. Exercise • New program in nanobiomolecular engineering • Define two program objectives.

  20. Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review

  21. Program Outcomes* • “…statements of what students are expected to know and be able to do by the time of their graduation.” • Outcomes must “…foster attainment of the program objectives…” • Process to produce outcomes • Assessment process, with documented results • demonstrating measurement • demonstrating degree of achievement • Evidence results used for program improvement *Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org

  22. Program Outcomes • Minimum outcomes “a-k” plus program criteria • Opportunity to be unique, i.e., define unique outcomes, not just repeat “a-k” • Helpful to map program-defined outcomes into a-k • Helpful to map outcomes into classes • Necessary to map outcomes into objectives

  23. Exercise • Define three unique program outcomes for the nanobiomolecular engineering program.

  24. Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review

  25. Performance Criteria • What will students do to demonstrate achievement of outcome • Example: “An ability to communicate effectively.” (3g in ABET list) • What are attributes of effective communication?

  26. Performance Criteria for Effective Communication • When making an oral presentation, students will • maintain eye contact • … • A written report will • follow a prescribed format • demonstrate proper grammar and punctuation • adhere to commonly accepted word usage

  27. Performance Criteria Learning Outcome 1.Has knowledge of current technological issues related to XXX engineering and society 2. Is able to discuss major political and societal issues and their pertinence to XXX engineering Educational Objective Contempo-rary Issues Professionally responsible

  28. Exercise • Define two or three performance criteria for one of the outcomes previously defined.

  29. Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review

  30. Indirect vs. Direct Assessment • Indirect – based mostly on student self-evaluation • surveys • interviews • focus groups • Direct – by faculty or some other means of evaluation of student performance (advisory boards evaluate design projects)

  31. Indirect vs. Direct Assessment • Indirect • necessary, but not sufficient provides uncalibrated snapshot • self-assessment not necessarily reliable • Terminology may be unfamiliar • Direct • necessary for quality assessment plan • not now for all, but new Program Evaluators are being trained to look for this feature • relies on faculty experience, expertise, and judgment

  32. How do we know if the students have the requisite outcomes? • When thestudentsthink they do, based on student survey results • With direct evidence from student work • Evidence is the key to accreditation • Faculty evaluation of student work is the key to providing evidence

  33. Assessment Measures • Primary assessment of student outcomes should be based on student work (direct) • e.g., student portfolios, student projects, assignments and exams, some employer surveys where skill is observed • Secondary evidence • Senior exit surveys, alumni surveys, employer surveys (qualitative evidence based on opinion), other • Combination of both methods – triangulation

  34. Indirect Assessment Measures • Surveys • Interviews • Course satisfaction surveys

  35. Direct Assessment Measures • End-of-Course Assessments • Targeted Assignments • Capstone Experiences • Capstone Exams • Portfolios

  36. End-of-Course Assessments • Course should have objectives • perhaps set by department committee • Related to program outcomes • Assigned problems (assignments, exams, projects) each related to course objectives • Evaluation/reflection by instructor

  37. End-of-Course Assessments • Advantages • quick and easy • assessment can be done in parallel with grading • Disadvantage • not comprehensive • no big picture • Opinion • a component of assessment plan

  38. Targeted Assignments/Problems • Key assignments that relate to specific program outcomes • Multiple assignments per outcome recommended • Integrate through curriculum • can demonstrate progress toward achievement of program outcome

  39. Targeted Assignments/Problems • Advantages • quick and easy • assessment can be done in parallel with grading • Disadvantages • none really • need consistent evaluation method with reliable inter-rater reliability

  40. Capstone Experiences • Can be design, laboratory, research • All programs have them • Where students are supposed to demonstrate and synthesize what learned • Usually includes teamwork, communication

  41. Capstone Experiences • Advantages • already required in program • assessment can be done in parallel with grading • Disadvantages • none really • need consistent evaluation method with reliable inter-rater reliability

  42. Capstone Exams • FE Exam • detailed, subject-related results available • Department-generated

  43. Capstone Exams • Advantages • FE is standardized • the most direct measure possible • Disadvantages • FE may not set bar as high as some want • students may not take department-generated exam seriously if results do not impact grades or graduation

  44. Portfolios • Collection of student work demonstrating outcomes • Must also evaluate the portfolio • Can also have students reflect on work in portfolio

  45. Portfolios • Advantage • comprehensive • Disadvantages • portfolio evaluation is additional work • need consistent evaluation method with reliable inter-rater reliability

  46. Other Direct Assessment Methods • Journals • Concept maps • Oral presentations with follow-up questions (like M.S./Ph.D. defense)

  47. Exercise • Select an assessment method to be used for direct assessment of program outcomes previously defined.

  48. Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review

More Related