Improving Learning from Web-Based Training Courses: Research Evidence

1. Improving Learning from Web-Based Training Courses: Research Evidence Dr. Traci Sitzmann Advanced Distributed Learning Co-Laboratory

2. ARE WE: Providing access to the highest quality education and training, tailored to individual needs, delivered cost effectively, anywhere and anytime? Realizing the ADL Vision Web-based Learning Home Field Online Gaming Distributed Simulation School In Transit Embedded Training Content Repositories Office

3. ADL Co-Lab Network - U.S. • ADL Co-Laboratory HubAlexandria, Virginia • Academic ADL Co-LabMadison, Wisconsin • Workforce ADL Co-Lab Memphis, Tennessee • Joint ADL Co-LabOrlando, Florida

4. ADL Global Partnerships Canada ADL Partnership Lab (Ottawa - DND) United Kingdom ADL Partnership Lab (Telford, England) In negotiation Korea ADL Partnership Lab (MOCIE) NATO/ PfP (26 + 20) Taiwan Latin America ADL Partership Lab ILCE – (13 Latin American countries) Singapore Australia ADL Partnership Lab (DEST) DEST Department of Education, Science and Training ILCE KOR Ministry of Commerce, Industry, & Energy TWN Ministry of Economic Affairs SGP Singapore Armed Forces THAI Ministry of Science and Technology Instituto Latinoamericano de la Comunicación Educativa NATO NATO Training Group PfP Partnership for Peace/ADL Working Group

5. Overview • Meta-analysis on the effectiveness of Web-based instruction (WBI) • Prompting self-regulation to improve learning outcomes during WBI

7. Purpose of Web-based Training Meta-analysis • Compare Web-based instruction (WBI) and blended learning (BL) to CI in terms of: • Their effectiveness for teaching declarative and procedural knowledge • Student reactions to training • Examine training design characteristics which influence learning from WBI

8. Participant Demographics • 19,331 trainees • 168 training courses • 113 undergraduate courses • 29 graduate courses • 26 corporate training courses • Average age = 24 years

9. Analyses • Hedges and Olkin’s (1985) procedure → mean corrected d effect d > 0 indicates WBI is more effective than CI d = 0 indicates WBI and CI are equally effective d< 0indicates CI is more effective than WBI • Subgroup procedure → test for categorical moderators • Sample size weighted correlations & hierarchical regression analysis → test for continuous moderators

10. Meta-Analytic Notional Example

11. Effectiveness of Web-Based Instruction Declarative knowledge d = .15* • WBI is 6% more effective than CI for teaching declarative knowledge Procedural knowledge d = -.07 • WBI and CI are equally effective for teaching procedural knowledge Satisfaction with training d = .00 • Trainees are equally satisfied with WBI and CI

12. Effectiveness of Blended Learning Courses Declarative knowledge d = .34* • BL is 13% more effective than CI for teaching declarative knowledge Procedural knowledge d = .52* • BL is 20% more effective than CI for teaching procedural knowledge Satisfaction with training d = -.15* • Trainees are 6% more satisfied with CI than BL

13. Effect of Trainee & Situational Characteristics • Is the effectiveness of WBI, relative to CI, for teaching declarative knowledge influenced by the following factors? • Instructional methods • Experimental design • Learner control • Human interaction • Practice • Feedback • Age and population (students vs. employees)

14. Meta-Analytic Notional Example

15. Richard Clark’s Theory Media are “mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes change in our nutrition.” -- Clark (1983) • Instructional methods include lecture, practice, examples, discussion, videos, etc. • Delivery media include CI, computer-assisted instruction, WBI, etc.

16. Similarity of Instructional Methods Hypothesis: WBI and CI will be equally effective for teaching declarative knowledge when similar instructional methods are used to deliver the two courses. • Same instructional methods d = .04 • Different instructional methods d = .29 • Instructional methods rather than delivery media predict learning outcomes

17. Experimental Design Hypothesis: WBI and CI will be equally effective for teaching declarative knowledge when studies used a true experimental design. • Experimental designs d = -.26 • Quasi-experimental designs d = .18 • Results support Clark’s position that media effects in single study research are largely spurious

19. Learner Control Learner control refers to the extent to which trainees have control over the content, sequence, and pace of instruction • Low learner control d = .07 • High learner control d = .30 • Trainees learn more when with a high than a low level of learner control during WBI

20. Human Interaction Human interaction indicates the extent to which trainees interact with the instructor and other trainees throughout the course • Low human interaction d = .19 • High human interaction d = .18

21. Practice & Feedback • Practice results • Only WBI incorporated practice d = .31 • WBI and CI incorporated practice d = .16 • Only CI incorporated practice d = -.27 • Neither WBI or CI incorporated practice d = -.25 • Feedback results • Only WBI incorporated feedback d = .33 • WBI and CI incorporated feedback d = .16 • Only CI incorporated feedback d = -.27 • Neither WBI or CI incorporated feedback d = .08

22. Population & Age Does learning from WBI, relative to CI, differ for students and employees after controlling for the age of trainees? • Age accounted for a significant 42.2% of the variance in declarative knowledge • Effectiveness of the WBI did not differ for students and employees after controlling for the age of trainees

23. Designing Effective Web-Based Courses • Incorporate a variety of instructional methods • Require active involvement of trainees • Provide an Internet skills course for trainees lacking technical skills • Provide trainees with control during training • Online communication should be synchronous rather than asynchronous • Trainees should practice the training material • Provide trainees with feedback throughout the course

25. Self-Regulation • Self-regulation is a process that enables an individual to guide his or her goal-directed activities over time and across changing circumstances (Karoly, 1993) • Gradual effect on learning

26. Self-Regulation Prompts Am I concentrating on learning the training material?   Do I need to continue to review before taking the final exam?

27. Self-Regulation Prompts Am I setting learning goals to ensure I have a thorough understanding of the training material? Do I understand all of the key points of the training material?

28. Self-Regulation Conditions • Immediate self-regulation • Prompted self-regulation after all 10 modules • Delayed self-regulation • Prompted self-regulation after modules 5-10 • No self-regulation • Did not prompt self-regulation

29. Study 1 • Online Blackboard training course • Learning was assessed with multiple-choice and performance exams at the end of training • Participants: • 93 trainees • 90% were instructors at a university or community college • 78% had a master’s degree, Ph.D., or M.D. • Average age = 44 years

30. Study 2 • Air Traffic Controller Task • Assessed performance in a simulation • Examined two dimensions of performance: • Basic • Strategic • Participants • 171 undergraduates • Average age = 19 years

31. Results

32. Results

33. Results

34. All Trainees Benefit from Prompts? • Cognitive ability • High ability trainees may be more capable of managing their own learning • Self-efficacy -- trainees’ perceptions of their ability to reach a goal • Trainees with high self-efficacy are more resilient when they receive negative feedback • Trainees with high self-efficacy may be more likely to use the prompts to adjust their training behavior

35. Results

36. Results

37. Results

38. Results

39. Results

40. Results

41. Conclusions • Prompting self-regulation improved learning outcomes • Trainees with high self-efficacy and high ability may benefit more from the prompts • Prompting self-regulation is a no-cost intervention • Can be reused in any web-based training course

42. Thank Youtraci.sitzmann.ctr@adlnet.gov

43. This is a good time to tell you that research has shown that asking yourself questions about whether you are concentrating on learning the training material will increase your performance on the test following training.  The training program will periodically ask you questions about where you are directing your mental resources and whether you are making progress towards learning the training material.  Your goal is to insert a goal relevant to the course content. Think about these questions and use your responses to decide how to allocate your review time. Am I concentrating on learning the training material?   Are there areas of training I am going to have a difficult time remembering for the final exam?

44. Are the study tactics I have been using effective for learning the training material? Do I know more about the training material than when training began?

45. Do I have thoughts unrelated to training that interfere with my ability to focus on training? Have I forgotten some of the ideas introduced in previous training material?

46. Research indicates that asking yourself these questions throughout training results in higher levels of performance in training. You should continue to ask yourself these questions as you study the training material. Am I setting learning goals to ensure that I will be ready to take the posttest? Would I do better on the final exam if I studied more?

47. Have I developed a strategy for increasing my knowledge of the training material? Am I making progress towards insert goal relevant to the training content?

48. Am I distracted during training? Have I reviewed the training material as much as necessary for the final exam?

49. Are the study strategies I'm using helping me learn the training material? Have I spent enough time reviewing to remember the information for the final exam?

50. Am I setting learning goals to help me perform better on the final exam? Do I need to continue to review before taking the final exam?