Outcomes Assessment in the Distance-Learning Environment

1. Outcomes Assessment in the Distance-Learning Environment

2. We are: Citrus College, Glendora, California (LA County) Cedar Valley College, Lancaster, TX (Dallas) • Marcy Morris: Learning Center, English (CC) • Theresa Villeneuve: Communications, Fine Arts • Tom Eiland: English (CC) • Lisa Nightingale: Instructional Support & Dist. Ed, Psychology (CVC) • Eric Nightingale: Psychology, History (CVC)

3. Citrus College, Glendora, California (LA County) 11,000 Students 100+ DE sections this semester As many as 4000+ students per semester Cedar Valley College (Dallas CCCD) 4800 students 160+ DE sections this semester As many as 4000+ students per semester

4. The Questions… • Do differences exist in outcomes assessment for the distance education / online environment vs. the on-ground, traditional environment? • Should they be different for the DE environment? • Citrus’ DE program & the DCCCD have been evaluating/reevaluating outcomes for 11 years.

5. DE Curriculum Outlines—The History… The first big questions… • 1996—discussion about whether DE courses should have a separate course outline • contact hours—connection with students?

6. More History… • 1998—incorporated the Distance Education course outline into the conventional course outline. • Independent/Directed Study added as method of instruction. • Instructor syllabus included which specifies how contact occurs.

7. The History, continued… • March 10, 1998—California Board of Governors approved recommendations to Title 5 Regulations on Distance Education. • Each DE course to be separately reviewed and approved. • Emphasis on regular effective contact

8. And Yet… Even More History… • 2004: Distance Learning Course Outline Addendum • Benefit of providing course in DE format • Methods of contact • Course objectives / outcomes / assignments / assessments • Methods of evaluation for online mode.

9. Accreditation Standards...a tilt…? • An effective institution maintains an ongoing, self-reflective dialogue about its quality and improvement (Introduction to the Accreditation Standards 2002)

10. But, That Was Not Enough... • Distance Learning Manual • published August 2006 by Accrediting Commission for Community and Junior Colleges (ACCJC) / Western Association of Schools and Colleges (WASC)

11. ACCJC / WASC Distance Learning Manual • Institution: Academic standards for all courses and programs should be the same as for all other educational experiences delivered by the institution. • Students: Students should expect that distance learning programs will permit completion of learning outcomes and objectives in the same manner as those delivered in traditional programs. • which validated our DE curriculum “journey”

12. Interesting Acknowledgement:methodology used in DE classes have been incorporated into classrooms on campus • Communication: • Email • Web • Discussion / message / announcement Board • Chat • Other: • Websites/FAQ’s • Course and Study Materials • Turnitin.com • Sample material

13. The Charge • To identify elements of quality online courses rather than elements of good instruction. • To identify class management standards and responsibilities for teaching online.

14. The Action • Review of existing course standards documents/models

15. Quality Matters: Inter-Institutional Quality Assurance in Online Learning A Grant Initiative of MarylandOnline Sponsored by the U.S. Dept. Education Fund for the Improvement of Postsecondary Education (FIPSE)

16. FIPSE Interested Because … • Quality assurance of courses is important • Voluntary inter-institutional assurance has never been done before • This can serve as a national model Quality Matters!

17. For Our Purposes, Quality Is… 100 Higher Standard • More than average; more than “good enough” • An attempt to capture what’s expected in an effective online course • Based on research and widely accepted standards 50 0

18. “They” say there is . . . .“No Significant Difference” • In student outcomes between alternate modes of educational delivery . . . but • The No Significant Difference Phenomenon(Thomas L. Russell2001, IDECC) • includes 355 research reports, summaries, and papers that document no significant differences (NSD) http://nosignificantdifference.wcet.info

19. there is a Significant Difference • When comparing face-to-face education with Internet-Based Distance Education, there is a significant difference in favor of the Internet-Based Distance Education . . . . In regards to student outcomes Significant Difference - Better Results with Technology 2006 - Sahin, C. S.Overcoming the "No Significant Difference" Phenomenon in Distance Education by Internet Andolu University, Turkey http://aof.edu.tr/iodl2006"...

20. Example: Freshman Composition 101: • 1996 : Created freshman reading and composition course for distance learning • Faculty demands: • Maintain course standards • Administration requirements: • Cannot change course outline • Only add Independent Study

21. Traditional Primary Learning Objectives Analyze Fiction Objectively Present Research Paper Demonstrate Use of Literary Terms in Deductive Essay Utilize MLA Formatting

22. Distance Education Objectives • Maintaining the Flexibility of Distance Education • Providing Students with Quality Educational Experience • Keeping Distance Ed Classes Legitimate in the Eyes of Academia

23. Traditional Assessment Tools • Timed, Proctored, Deductive Essay • Analytical Research Paper • In-class Lecture • In-class Discussion • Topical Presentation

24. Primary Hurdle • Primary concern among English faculty when presented with possibility of distance education Freshman Comp online course was synchronous class contact. • Theory was that in order to understand the application of literary terms in the analysis of fiction, one must be able to practice through class conversation

25. Other factors . . . . • Ensuring rigidity of testing • Time • Security • Content • Format • Maintaining quality and quantity of written research essays as required by established college standards and IGETC agreement (Intersegmental General Education Transfer Curriculum)

26. Solutions… • Classroom time / Synchronous Contact • Replaced with weekly chat room meetings • Synchronous conversation with students • Chat room transcript archived on web site for review and retrieval • All students required to attend or to read archive and respond

27. Solutions…continued • Testing Protocol: • Establishment of Proctored Testing Center • Testing center controls: • Allowed materials • Allotted time • Identification of test-taker • Adapted Testing needs

28. More Solutions • Term Papers: • Strict adherence to established guidelines • Content • Length • Format • Peer editing asynchronously through message board • Turnitin.com • SafeAssign

29. Solutions… (almost done) • Asynchronous Contact • Message board • E-mail • Web page • Archived chat room • Web links

30. Results • Comparing ENGL 101 and ENGL 103 Freshman Composition Courses • Retention is higher in on-campus courses • Traditional Avg: 22.5/30 • Distance Learning Avg: 15.3/30

31. Results…continued • Grade range reflects a standard curve in traditional courses (frown) • Traditional Delivery Averages: • A: 18% • B: 36% • C: 24% • D: 22%

32. More Results… • Distance education grade range is wider (smile) • Distance Education Grade Averages: • A: 38% • B: 23% • C: 8% • D: 31%

33. FACTORS (as determined by observation, surveys and interviews) • Population • Age (older is better) • Experience (older is still better) • Distractions (less is better) • Motivation (more is better) • Delivery Method • Technology (BB, attachments, chat room) • Influence of real, live person

34. What It Means . . . . • Delivery methods create some different outcomes BUT… • Assessments tools and criteria are the same • Testing center more uniform than individual classroom practices (now being used for traditional courses as well) • Technical Support would help many students succeeded in distance education • The distance education version of freshman composition at Citrus College has as much, if not more, student learning outcome assessment and verification than the on-campus version of the same course.

35. Art 199 Motion Picture Appreciation Distance Ed vs. Traditional Deliveries

36. Content Of The Course • Social/technical factors of film production • Genres • Aesthetics • Narrative and non-narrative construction • Film form and history • Ability to make sophisticated small talk at cocktail parties.

37. Concerns • Logistical concerns re: films • Rigor • Quality and Quantity of education to maintain IGETC agreement • I am the only faculty member who teaches this course.

38. Methods of Instruction(from course outline) • Lecture • Lecture/Discussion • Independent Study • Out of class film viewing • Directed study

39. Assessment • Discussion • Large Paper • Quizzes • Taken online • Tests • Taken in testing center under controlled conditions until this semester

40. What it means • No one flies under the radar • No one can sleep through a film and pass my class • Testing is controlled • Criteria are unchanged • At least the same learning outcomes as a traditional course.

41. Overall Midterm Comparison, Fall 2006

42. Final Comparison, Fall 2006

43. CVC Online Success & Retention

44. CVC Spring 2006 Distance Learning and On Campus Grade Distributions

45. CVC Spring 2006 Distance Learning and On Campus Grade Distributions

46. What Makes DE Work? . . . INFRASTRUCURE! A viable Distance Learning program requires changes to the infrastructure of campus to sustain learning outcomes mandated by: • College District • State Regulations • Accrediting Commission • Federal Government—Title V

47. Infrastructure ChangesTesting Center • Grew out of need to offer timed, proctored testing for DE students to establish controls for test taking to insure quality • Meets regular effective contact requirementsto authenticate outcomes • Now serves DE as well as the larger campus community--academic makeup testing, assessment testing, adapted testing accommodation, job applicant testing, computer skills competency testing.

48. Infrastructure—Supporting Services of SLO’s in DE • Distance Education Department • Administrative, Clerical, and Technology Support Staff • Training for Students and New DE Instructors • Orientation of New Students and New Faculty • Online Services including registration and tutoring • DE Student Support staff (between technical & counseling)

49. The End? • DE was implemented long before and is keeping pace with if not exceeding SLO requirements for traditional courses • Distance Education has clearly prompted the college to reevaluate outcomes, has affected how we teach in the traditional classroom, and has effected a change in infrastructure to meet needs in regards to outcomes.