Applying the QM Rubric

1. Inter-Institutional Quality Assurance in Online Learning Applying the QM Rubric August 2008

2. After this training you will be able to… • Describe underlying principles of Quality Matters (QM). • Use QM rubric& explainscoring system. • Apply the QM rubric to online course and scenarios.

3. QM Materials Overview • Today’s Agenda • Folder Overview • QM Website www.qualitymatters.org • QM Contact Information

4. Introductions • Share name, institution, job, and best distance learning practice. • Briefly (in one sentence) describe your best distance learning practice.

5. Trying it out with Standard 1.2 • A statement introduces the student to • the purpose of the course and to its • components; in the case of a hybrid • course, the statement clarifies the • relationship between the face-to-face • and online components.

6. Review Scenario This course reflects a shift in the importance that the world outside the schools increasingly places on thinking and problem solving. Procedural skills alone do not prepare students for that world. Therefore, students deserve a curriculum that develops their mathematical power and an assessment system that enables them to show it. Assessments that match the current vision of school mathematics involve activities that are based on significant and correct mathematics. These activities provide all students with opportunities to formulate problems, reason mathematically, and make connections among mathematical ideas. Students engage in solving realistic problems using information and the technological tools available in real life. Moreover, skills, procedural knowledge, and factual knowledge are assessed as part of the doing of real life mathematics. 1. 5-Tests (Must be taken on campus) 400 2. 15-Quizzes 75 3. Project (Optional) 50 Total number of points possible 475 (no project) 525 (if you do the project) Grading Scale: 90% to 100% = A 80% to 89% = B 70% to 79% = C 60% to 69% = D 0% to 59% = F A student who earns a final average of 70% or more on all assessments of objectives and intended learning outcomes for the College Algebra course has successfully fulfilled the general education and other essential core skill goals. Any extra credit problems are given to the entire class. There are no extra credit problems for any one individual during any part of the semester. You are reviewing a College Algebra course and find the following statement:

7. About Quality Matters

8. Course Meets Quality Expectations Course Revision Quality Matters: Peer Course Review Process Faculty Course Developers Institutions National Standards & Research Literature Course Rubric Faculty Reviewers Training Peer Course Review Feedback Instructional Designers

9. QM = Rubric and Process Rubric Process • Outcome: earn QM recognition • Tool to assess online courses during formal QM review • Outcome: improve courses, meet institutional goals • Guide to develop new online courses AND review and update online courses • Outcome: earn QM recognition • Must follow official QM guidelines and procedures • Outcome: improve courses, meet institutional goals, demonstrate commitment to quality

10. Factors Affecting Course Quality • Course designQM REVIEWS THIS • Course delivery (i.e. teaching, faculty performance) • Course content • Course management system • Institutional infrastructure • Faculty training and readiness • Student engagement and readiness

11. Design vs. Delivery The faculty member is integral to both design and delivery. Course Design …is the forethought & planning that a faculty member puts into the course. Course Delivery …is the actual teaching of the course, the implementation of the design. QM is about DESIGN - not delivery or faculty performance

12. Distinguish between design & delivery… Example: Discussion Board Design: Discussion boardplanned in course; students told how they should participate & how they can expect the faculty to participate. Delivery: How often the faculty member actually participates in the discussion; what the faculty member actually says to students.

13. For Our Purposes, Quality Is… • More than average; more than “good enough” • Attempt to capture what’s expected in an effective online course at about an 85% level • Based on research & widely accepted standards 85 %

14. What QM is NOT About… • Not about an individual instructor • (it’s about the course) • Not about faculty evaluation • (it’s about course quality) • Not about judgment(it’s about diagnosis and improvement) • Not about “win/lose” or “pass/fail” • (it’s about continuous improvement in a supportive environment)

15. QM Collegial Review vs. Faculty Evaluation QM Review Faculty Evaluation • Single point in time • Focus: delivery • Outcome: decision on performance for promotion/tenure • Characteristics • Win/lose situation • Confidential/secretive • Ongoing • Focus: course design • Outcome: course improvement • Characteristics • Non-threatening • Team approach that includes faculty • Full disclosure to faculty

16. The Rubric

17. The Rubric • Eight General Standards: • Course Overview and Introduction • Learning Objectives (Competencies) • Assessment and Measurement • Resources and Materials • Learner Engagement • Course Technology • Learner Support • Accessibility Key components must align. Alignment:Critical course elements work together to ensure that students achieve the desired learning outcomes.

18. Rubric Scoring In an official QM Review: • Team of 3 reviewers initially score individually • One score per standard based on team majority • Pre-assigned point value • Yes/No decision; All/None points • Consensus is NOT required

19. Points are NOT on assigned on a sliding scale…

20. To Meet Expectations…Two Thresholds • A course must achieve: • “Yes” on all 17 of the 3-point “essential” standards. • A minimum of 72 out of 85 points

21. The Instructor Worksheet

22. Instructor Worksheet • Read it first • Refer to it during the review • Use in team discussions

23. Instructor Worksheet = The Voice of the Instructor • Key piece in the review • Includes info about: • Institutionally mandated objectives, materials, practices, policies • Materials outside course pages • Types of interaction used & instructor’s statement on the appropriateness of interaction in the course • Additional items that may require review

24. Hands On Practice

25. Your Point of View as a QM Course Reviewer… • Take the students’ point of view • Advocate for the student • Can’t find evidence standard is met…don’t assume it is or isn’t there….ask the course developer (faculty member)

26. How to decide…. For EACH standard • Read the specific review standard and the annotation; review the examples, if needed. • Look for evidence that the standard is met in this course. • Ask yourself: Does this course meet the standard at an 85% or better level? • Decide Yes or No and enter your answer in the web-based rubric form. • Include comments/suggestions as documentation.

27. Hands-On Practice • Review the Anthropology Course • Go to http://pgcconline.blackboard.com • Username is fipse • Password is fipse • Focus on 17 essential (3 point) standards • Write 1 recommendation per standard

28. Hands-On Practice • Find a Partner • Locate login handout in folder • Login to ANT 103 • Read Instructor Worksheet for course

29. General Standard 1: Course Overview and Introduction

30. What Are the Consequences of Your Decision? • In a formal QM review: • What does a YES decision mean? • What does a NO decision mean?

31. Recommendations

32. No Choice! A recommendation MUST be written if you decide that the course does not meet the specific standard.

33. How to Write a Useful Recommendation

34. Improve these recommendations… • Assignment instructions weren’t clear. • I wasn’t sure if assignment #1 required a written paper. • You didn't tell the students how to find the additional resources. • I had trouble locating the resources; you may want to put a link on the home page for easy access. • The text on the page was too hard to read. • The green text on a blue background was difficult to read. You might want to try basic black on white.

35. The learning objectives aren’t measurable. • I had trouble figuring out how the objectives would be measured. I think it would help if you rephrased the learning objectives using active verbs (explain, distinguish, compare, etc.) The assessments are weak. • I had difficulty connecting the assessments to the learning objectives; they seemed unrelated to the learning objectives. I think you’ll want to review the learning objectives and make sure that they are reflected in the exams.

36. Apply the QM Standards To the ANT Course (continued)

37. General Standard 2: Learning Objectives (Competencies)

38. Course Learning Objectives (2) Learner Interactions & Activities (5) Resources, Materials (4) & Technology (6) Assessment and Measurement (3) Key sections that must align…

39. Remember Bloom?

40. Think About “Alignment” From an Introduction to Psychology Course Objective: Summarize the relationships that exist between biology and human behavior

41. General Standard 2:Learning Objectives (Competencies)

42. General Standard 3: Assessment & Measurement (we’ll come back to this one) 3.1: The types of assessments selected measure the stated learning objectives & are consistent with course activities and resources. 3.2: The course grading policy is stated clearly. 3.3: Specific and descriptive criteria are provided for the evaluation of students’ work and participation.

43. General Standard 4: Resources & Materials

44. General Standard 5: Learner Engagement

45. General Standard 6: Course Technology

46. General Standard 3: Assessment & Measurement

47. General Standard 7: Learner Support • No “essential” 3-point elements • Primarily concerned with instructional support – an institutional responsibility

48. General Standard 8: Accessibility

49. Scenarios

50. Evaluating Scenarios • Divide into groups. • Discuss the scenarios that focus on the 17 essential standards. • Take the “quiz” with your group. Use the material in your books and each other as references. • Jot down your reasoning and be prepared to discuss your decision.