TELOS TESTBED – 2006Evaluating Learning Design Tools Karin Lundgren-Cayrol Isabelle Savard Marcello Maina Claire Banville En collaboration avec: Theme 6.3 France Henri & Josianne Basque
Outline • 2006 - Testbed Context, Goals and Objectives • General Evaluation Plan & Formative Evaluation Design • Participants • Hands-on Evaluation Scenario • Results • Background questionnaire • Testbed Evaluation Scenario • Final Questionnaire • Preliminary conclusions • Connected Student Thesis Work • Isabelle Savard (Poster78 : Cultural Diversity and Reuse of Learning Objects) • Marcelo Maina (Adapting MISA to IMS Learning Design) • Questions
Goal • To identify strengths and weaknesses of the TELOS Instructional Design Interface potentialTools
Objectives 2006 • To test tools that are scheduled to be incorporated into the TELOS instructional design Interface. This year: • MOT+LD editor • Canadian LD Repository • Learning Design Methodology • To elicit information about practitioners ways and needs in terms of facilitating their instructional design tasks. • To send results to concerned teams about their tools in regards to their integration into TELOS. • Team 6.3 – ID’s opinions, tasks and perceived competency profile • Team 2.1 and 6.2 – Revisions of processes, tools and guides
demands demands demands Identify needs Create testbed scenario Create testbed protocol Carry out testbed Analyse results Protocol & Testbed Cycle Revise TELOS Revise LKMS Revise LKMA Revise LKMA Core LKMS LKMA LKMP Disseminate
Context 2006 • TELOS consists of four different interfaces and toolsets according to actor • Engineer (TELOS Kernel) • Technician (LKMS) • Instructional Designer (LKMA) • Learner and Teacher Interfaces (LKMP)
Call for Participants • Requirements: • Experience in designing online learning / e-learning • Good computer knowledge (Windows) • Experience with web-based technology
Testbed Agenda & Instruments • Complete Background Information concerning (15 min) • Work Context • Technical Profile and Tool Usage • ID Experience and Practice • Carry out the Evaluation Scenario • Demo Session (1h30) • Hands-on (5 h) • Complete a Final Questionnaire (30min) • Participate in a Focus Group (1h)
Let’s visit the IDLD siteAnd carry out this EVALUATION Scenariohttp://www.idld.org
Completion Rates • Background questionnaire (N=11) • 9 Mauritians (Validation of instrument) • 11 Canadians • Demo session (N=9) • 1 in LORIT; 6 = ClickToMeet; 2 by LORIT Webcast & chat; 1 used • 2 abandon • Evaluation Scenario (N = 6 - 1) • 2 abandon; 1 finished but did not return it; • 3 about half; 1 all but adapting UoL; 1 all the tasks • Final Questionnaire (N = 6) • 4 most questions • 2 less than half • Focus Group = cancelled
Background Information - Context • Mauritius (N=9) • F2F interviews allowed validation of BQ • All worked at the University (7 ID; 2 tutors) • 3 designed in English, 2 in French, and 4 in Both languages • 5 had LO repository access • Canada (N=11) • Geographically distributed (contact by email and phone) • 8 University, 2 College and 1 Government • 4 Prof/Teach; 5 ID; 1 Tutor ; 1 Training Manager • 7 designed in English, 1 in French, & 3 Both languages • 2 LO access; 2 under development; 6 NO &1 didn’t know
Context (2) Tendency:Instructional designers work in any discipline, teachers referred to their domain expertise.
Background Information – Canadians Tool Usage • 10/11 mostly used Windows • 6/11 used IE, 4 Mozilla & 1 Netscape
Preferred Help Method Conclusion: Telos must provide a plethora of advisory systems as well as having human resources available through the use of sophisticated communication tools.
Tool Usage (3) LO and Repository Conclusion: Most LO, IMS LD related tools are not widely known.
Background Information – ID Experience and Practice (1) • Canada • 5/11 had more than 6 years • 3/11 had between 3 and 6 years • 3/11 less than 2 years • Mauritius • 1/9 had more than 6 years • 7/9 had between 3 and 6 years • 1/9 less than 2 years 16/20 more than 3 years Conclusion: TELOS must be able to provide flexible instructional design environments.
Reusing LO’s Conclusion: LO’s are used in different ways and are starting to be a well known concepts.
Validation process and results Validation process in Mauritius (Savard, 2006) • First version of the background questionnaire • Semi-structured interviews to find out their regular ID process and tasks • Similar to Gagné’s ADDIE Instructional Design Model. Results: • Comment on each question which led to • changes to Que’s 5, 10 & 21 • addition Que’s 17, 19 & 22 (ADDIE)
ADDIE - Instructional Design Task • Analysis • Design • Development/Production • Implementation • Evaluation
ADDIE - Instructional Design Task • Analysis Conclusions • In general, the Analysis task is already done when a demand comes to Mauritians Instructional Designer’s. • They often have to reformulate objectives. • For the Canadians, the Analysis is an important task.
ADDIE - Instructional Design Task • Design Conclusions: • Instructional Structure is an important task for both ID from Mauritius and Canada.
ADDIE - Instructional Design Task • Development/Production Conclusions: • Material production is an important task in both Mauritius and Canada. • In Mauritius the delivery and maintenance plan seem not to be their responsibility.
ADDIE - Instructional Design Task • Implementation • Conclusions: • Small field testing doesnt take too much time for ID from both contries.
ADDIE - Instructional Design Task • Evaluation Conclusions • The course evaluation is not a major concern in Mauritius. • 70% of Canadians do carry out this task
ID Competencies (ibstpi) • Four groups of Competency statements for instructional designers: • A- Professional Foundations • B- Planning and Analysis • C- Design and Development • D- Implementation and Management • Validate which competencies were the most important in their job situation. http://www.ibstpi.org/
Comments and Suggestions • Experience too time consuming because of all the documents to be studied • Several separated Demo Sessions are necessary • IMS LD very complex • MOTPlus has significant advantages • Graphical modeling is interesting • All tools were very interesting
IDLD PORTAL Conclusion: Very divided opinions, we need more evaluators!!
MOT+LD and Graphical Modeling Conclusion: At least useful and functional for those who tried
PALOMA Resource Manager Conclusions: Divided opinions and Needs more evaluators!! • Suggestions: • Need a more user-friendly version • We need more reliable federated search protocols • Advanced search is a bit confused
Some Preliminary Conclusions • The ADDIE Model and ibstpi competency • Generally practiced by ID’s (few differences) • Questions whether Planning and Evaluation is an ID task? • TELOS must be able to provide • Sophisticated communication tools to simulate human resource contact • flexible instructional design environments • Confirms complexity of the IMS LD concepts • Need to adjust vocabulary • Provide transparent & flexible tools • Provide more examples in the Canadian LD Respository