Executive Beliefs About Critical Success Factors:

1. Executive Beliefs About Critical Success Factors: Defining, Designing, Developing and Deploying eLearning Ann Armstrong, Executive Director The Center for Educational Outreach and Innovation (CEO&I) President, TC Innovations Teachers College Columbia University November 9, 2006 Email:aarmstrong@tc.columbia.edu

2. Agenda • Background for Study • Study Design • Conceptual Framework • Synthesis • Findings • Synthesis Applied to Case #2 • Conclusion

3. What Leaders Are Saying • “The biggest growth in the Internet, and the area that will prove to be one of the biggest agents of change will be in eLearning.” - John Chambers, CEO Cisco Systems • “eLearning has lots of promise, but it takes commitment and resources for an organization to see its potential realized.” - Allison Rossett, Professor of Educational Technology at San Diego State University • “Whether you are wading eagerly in the eLearning ocean for a bracing swim, or pacing the shore looking for the lifeguard, you cannot ignore the impact of eLearning. It is the transformational event of our professional lives.” - Tina Sung, President and CEO of the America Society for Training and Development

4. Research Facts • Researcher Perspective • 25 years Business and Academic Experience in Technology Based Learning • Executive Director, The Center for Educational Outreach and Innovation, Teachers College Columbia University • Research Problem • eLearning Technology is Evolving at Incredible Speeds • eLearning is Transforming Learning Delivery • There is a Dearth of Research Available on How to Deliver Effective eLearning • Lack of Comprehensive Well Structured Studies

5. Research Facts • Research Purpose • To garner an understanding of what eLearning executives learned from the experience of defining, designing, developing, and deploying large-scale, web-based learning programs for adult professional development in major corporations. • To uncover what factors these executives believe to be critical and necessary for the organization to successfully deliver training and education over the world wide web.

6. Design: Qualitative, Multi-person, Mixed Method

7. Conceptual Framework

8. Synthesis: Eight Principles for Success: eSUCCESS executive Sponsorship Support from the Organization Understand and Motivate the Learner Culture Fosters Learning Content is Relevant to Learner & Organization Evaluate and Assess Structure of Program is Engaging, Interactive and Blended Simulate the Work Environment and Work Tasks

9. Analytic Category 1.0: Support – Case Description

10. Support • “There is no one in the company more influential than front-line managers - they can make organizational change or kill it outright.” • Marc Rosenberg, 2001 • “If the manager takes it seriously, people will take it seriously, partly because it is the manager who does the performance reviews.” • VP Prestigious Leading Global Investment Bank • “Identifying, harnessing, and supporting the power of leadership are critical success factors for your eLearning implementation.” • Jay Cross and Lance Dublin, 2002 • “Well, I am certainly one of the key sponsors in that I run The Learning Council. So I am responsible for setting direction, getting funding for the infrastructure, getting buy-in to standard practices, polices, procedures and then also from the enterprise learning perspective, since I roll out a big chunk of the eLearning, I am responsible for funding too. So I am a huge sponsor.” • CLO World Top 3 Commercial Bank. Top 10 2005 Forbes Global 2000

11. Analytic Category 2.0: Culture – Case Description

12. Culture • “Organizational cultures and their leaders are reflections of each other. Building a culture that will embrace eLearning means building senior management support for that culture.” • Mark Rosenberg, 2001 • “We said to ourselves, you have to understand a level of safety awareness on all of our projects so this wasn’t just pushed down to 3 or 4,000 top notch people. It was also pushed down to well in excess of 10,000 individuals who work for our subcontractors. So now instead of counting deaths on an annual basis we are really counting saves, how many people we save. So the number of fatalities has gone down significantly. And, you can’t really put a price on human life. Safety has been very very instrumental. So that is probably the biggest thing. We really changed the culture within the organization as it relates to safety awareness, how you practice it, and how you apply it across all of our jobs.” • SVP Large Building Services Firm, Construction Industry

13. Analytic Category 3.0: Motivation – Case Description

14. Motivation • “Motivation, that is the drive of the learners to start and finish a course and transfer the knowledge back to the worksite results from the intrinsic drive within the individual learner and extrinsic efforts supplied by the online learning environment.” • Jim Moshinskie, 2002 • “They need to know this stuff to do their job. And this stuff happens everyday in their job, so this is what they had to know. The “what’s in it for me” was if you want to do your job better, then you need to do this. Or actually even more strongly if you want your job, you need to do this.” • Managing Director One of the Four Largest Banks in the World 2005 Forbes Global 2000 • “Positive support from management through rewards and incentives is also a key enabler to improved learner motivation.” • Allison Rossett, 2002 • “It is about what you are hearing on the street. You know topics come and go, topics are more or less relevant and I think there is particularly now that people are so inundated with the media and the like. That those things which seem timely and have a spark of recognition about them probably have a shorter curve to climb to get people to give them a shot.” • VP Prestigious Leading Global Investment Bank

15. Analytic Category 4.0: Program Design – Case Description

16. Program Design - Blended • “Are you getting the training that you need? I am tired of asking that question. You know because, we give them the training they need, but do they take advantage of that? Are they motivated to take advantage? So blended learning I think gets at the motivation issue a little bit deeper than straight online learning.” • CLO World Top 3 Commercial Bank - Top 10 2005 Forbes Global 2000 • “I think the blended factor was very significant because when you look at a topic like ethics there are some things that are very clear and some things that are very grey, and the interpretation of it is really very important for people to say, I thought this was ok, but now I see by this particular policy it is not and I had a situation very similar last week and I wasn’t sure how to handle that. So I think that is probably more, that discussion brings out new thoughts or ideas about how to handle that the right way.” • VP Top Pharmaceutical Company – Forbes 500

17. Program Design – High Quality, Interactive, Engaging • “Innovation in terms of the journey through the lesson. Is it visual? Can you relate to it? Bringing it back to adult learning principles you know where people bring their experience to the learning activity and you are supporting the things they already know and building upon that rather than trying to force in content without really understanding the context. So I think that innovation is a big part of it, building the context so that they know what they are learning and why, giving it to them in a way that is meaningful, and exciting and interactive. Reinforcing it so that they don’t feel like they are being quizzed or tested, but it is an exciting way to apply the knowledge. Giving them a good, failsafe environment to transfer it back to the workplace. And those are the things I think where innovation comes in. It is how you express all that in an artistic way. It gets past the science and probably a little bit into the art.” • AVP One of the World’s Largest Insurance Companies, PHD. • “Online learners cannot stay motivated by turning pages on the web…Courses should include a variety of interactions, such as puzzles, case studies, scenarios and simulations that directly engage learners. • Jim Moshinskie, 2002

18. All Analytic Categories: Critical Incidents

19. Technology Infrastructure • “The low point quite frankly,. is the technology, and this may be something or an issue that everyone has to deal with, not just within our company, but it just seems as if it is always something. If it’s not speed, it’s reliability. If it’s not reliability, it is speed. If there is not a bug that pops up in the testing engine, there is a bug that pops up in the course itself.” • SVP Large Global Financial Services Firm

20. Case #2 Mark at Wakefield Jones • 13,000 Financial Advisors • Geographically Dispersed Across the United States • Different Levels of Knowledge and Experience • “Sales” Focused • Multiple Product Lines to Sell • High Net Worth Clients

21. eXecutive Sponsorship-Principle #1 • “Built into the Organization” • Executive Offices • Senior Management Support • Time • Money • Communications • Change Management

22. Support – Principle #2

23. Understanding and Motivating the Learner – Principle #3

24. Sponsorship, Support, Motivation • “We never imagined more than maybe 1,500-2,000 people going through this program, so to get up to 7,000 is really remarkable; and I think it shows if nothing else a thirst from people to be educated and I think people really appreciate the fact that it is at their fingertips and I think people really appreciate the fact that they are able to go through a quality program without leaving the office. So whether you want to credit it towards the fact that we reimburse upon successfully completing the program; whether you want to credit it towards the fact that we received a tremendous amount of support from senior management; whether or not you want to credit the fact that in some areas branch managers’ compensation was tied to their numbers within their offices, it could be a variety of reasons but to get 7,000 we would have never imagined that. So we’re pretty pleased.” - SVP Large Global Financial Services Firm

25. Culture – Principle #4 Systems Continuous Learning Sharing Learning Connecting to the Environment Learner Collaboration and Team Learning Promote Inquiry and Dialogue Leader Collective Mission

26. Content Relevance – Principle #5 • Current Job • Future Job • Industry Focus • “Branded to the Culture” • “Hot Industry Topics” • Part of a Continuing Education, Certification and/or Accreditation Requirement • Client Focus • Designed by Respected Subject Matter Experts

27. Evaluation and Assessment – Principle #6 • Built Into the Courseware • Passing Score Required • Kirkpatrick's Four Levels • Reaction • Learning • Behavior • Results

28. Blended Learning – Principle #7 OJT Coaching/Mentoring Traditional Classroom

29. Simulate the Workplace – Principle #8 • Workplace Scenarios • Work Tasks • Industry Specific Examples • Case Studies • “Caselette” • Branded to the Culture • People • Places • Things • Branding Standards

30. “The question is no longer whether organizations will implement online learning, but whether they will do it well.” Mark Rosenberg, 2001

31. Building Blocks for eSuccess SIMULATE THE WORKPLACE Case Studies “Caselettes” Branded TV Culture STRUCTURE Blended ILT OJT Coaching Engaging Innovative Visual Auditory EVALUATE AND ASSESS Built into Courseware Passing Score Required Kirkpatrick Levels 1,2,3,4 SUPPORT eLearning Leadership Executive Line of Business Direct Manager Involvement UNDERSTAND AND MOTIVATE THE LEARNER Relevant Content Certification Benefit Communicated Performance Plan “Peer Pressure” CULTURE Continuous Learning Systems to Share Learning Collaboration and Team Learning Inquiry and Dialogue Collective Vision Connecting to Environment CONTENT Relevant to Job Specific to Culture Purposeful High Quality, Enjoying Blended Case Studies Reputable SME executive Sponsorship “Built into the Organization”

32. Emerging Themes - Study • Executive Sponsorship • Support • Degree Organization’s Culture Supports Individual, Group and Organizational Learning • Relevance of Content to the Learner • External Motivation Factors

33. Emerging Themes – Literature • Culture, Champions, Communications, Change (Rosenberg, 2001) • Learner Motivation (Moshinskie, 2002;Schank, 2002) • Executive Sponsorship (Moshinskie, 2002; Hall, 2002) • Use of Goal-based Scenarios (Schank, 2002) • Measuring Results (Hall,2002)

34. Research Questions • How do different forms of support contribute to the success or failure of online learning program delivery? Support such as the following: • Executive Line of Business sponsorship • Executive e-Learning Leadership • Direct manager involvement • Participation in and recognition by a Community of Practice • Technology infrastructure • Other • How do specific aspects of a learning culture support the rollout and success of an e-Learning initiative? • How are external incentives, rewards and other learner motivational strategies perceived and used by the e-Learning executive to influence learner participation in a program and/or the learning garnered by adults in a web-based learning program? • How important is the nature of the content and its relevance to the learner’s job?

35. Study Design – Case Study • Qualitative • Multi-person • Mixed Method

36. Study Context - Organizations

37. Study Context – Organizations cont’d.

38. Study Phases cont’d. • Phase III: Data Collection • Gather extant Data • Complete Extensive Case Interviews • Code Data • Write Case Reports • Preliminary Analysis of Case Reports • Phase IV: Data Analysis, Synthesis & Reporting • Analyze and Synthesize Data (within and across case) • Adapt Theory • Adapt Conceptual Framework • Write Dissertation • Defend Dissertation

39. Data Sources • Pre-Inventory Data Inventory • Demographics • Survey data • Interview Questions • Interviews • Observations • Archival Meeting Minutes/Notes • Public Corporate Data • eLearning Course(s) • Dimensions of the Learning Organization Questionnaire (DLOQ)

40. Study Phases • Phase I: Define • Preliminary Literature Review and Conceptual Framework • Development and Selection of Data Collection Tools and Instruments • Selection of Pilot Case • Development of Preliminary Code List • Pilot Study • Analysis of Pilot Case • Refinement of Tools, Conceptual Framework and Preliminary Code List • Phase II: Design • In-depth Literature Review • Completion of Dissertation Proposal • Department Approval of Dissertation Proposal • IRB Board Approval • Final Select and Confirmation of 7 Cases

41. Conceptual Framework

42. Conceptual Framework

43. Conceptual Framework

44. Conceptual Framework

45. Conceptual Framework

46. Conceptual Framework

47. Building Blocks for e’ Success SIMULATE THE WORKPLACE Case Studies “Caselettes” Branded TV Culture STRUCTURE Blended ILT OJT Coaching Engaging Innovative Visual Auditory EVALUATE AND ASSESS Built into Courseware Passing Score Required Kirkpatrick Levels 1,2,3,4 SUPPORT eLearning Leadership Executive Line of Business Direct Manager Involvement UNDERSTAND AND MOTIVATE THE LEARNER Relevant Content Certification Benefit Communicated Performance Plan “Peer Pressure” CULTURE Continuous Learning Systems to Share Learning Collaboration and Team Learning Inquiry and Dialogue Collective Vision Connecting to Environment CONTENT Relevant to Job Specific to Culture Purposeful High Quality, Enjoying Blended Case Studies Reputable SME Executive Sponsorship “Built into the Organization”