WP4 – Pilot implementation Ongoing activities May 2012

1. With the support of the Life Long Learning Programme of the European Union WP4 – Pilot implementationOngoing activitiesMay 2012 Nathalie Younès, Christian Lallias, Lysiane Lelue

2. WP4 – Pilot implementationChoice of pilot organizations and processes 2 different/opposite organizations 1 overall objective in compliance with EMOLL project a similar general objective for each organization : efficiency/quality various specific objectives and uses PARALLEL PROCESS + INTERACTION and MUTUAL GOOD PRACTICES (but 2 different presentations) May 2012

3. WP4 Implementation and exploitation of EMOLL methodology at CAFOC Auvergne With the support of the Life Long Learning Programme of the European Union

4. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne What is CAFOC Auvergne CAFOC Auvergne (Centre Académique de Formation Continue) is part of an organization named DAFPIC-GIP that leads and manages the vocational activities of the French National Education at regional level (Auvergne region). CAFOC is part of the GRETA network in charge of the continuous training of adults (salaries and job seekers).

5. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Main fields of activities • Its main fields of activities are: • training of trainers and training advisors • training engineering • production of training tools (mainly online interactive tools) CAFOC is on a regional level but GRETA is a national network.

6. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne CAFOC motivation CAFOC Auvergne uses already evaluation processes and procedures (DAFPIC-GIP is ISO 9001 certified). It has analysed it’s work processes, broken down in activities correlated with the requested competences, which is always the basis of a quality policy. But EMOLL project is an opportunity for CAFOC staff to develop and deepen its evaluation processes, notably for the measurement in the medium-term of the actual results of its training courses. That is why CAFOC is particularly interested in the measurement and evaluation of the short and long term effects of training.

7. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Implementation planning 9 May meeting January March 2012 2 April meeting -highlighting and clarification of some concepts (perspective, strategic axis, mesure, target value, calculation procedure of results, ...) - work on KPI sheets. - recall of EMOLL objectives- recall of the main concepts of BSC- presentation of examples of perspectives, mesures, indicators (KPIs). prior contacts with CAFOC Internalbrainstormings 24 May meeting 13 June meeting making of a Simulator Guidebook in French presentationof the HR Scorecard simulator - validation of the KPIs sheets achieved in the meantime. - work on processes and tools to collect the results in a scoreboard. achievement of KPIs sheets by CAFOC application evaluation

8. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Strategic axis (updated 9 May meeting) • Measuring the results of external long-period trainings in middle-term Client • Measuring the results of internal long-period trainings in middle-term (in case of precise order from a Greta to meet an identified need) Client • Grading skills tranfer relative to « tool » trainings Client • Grading training actions implemented after analysing training needs of the employees within Greta Internal process • Measuring satisfaction « on the ground » of the trainees Training • Monitoring ressources employment (return on investment) Finances

9. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Adaptation of EMOLL tools in French

10. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Adaptation of EMOLL tools in French KPI sheet

11. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Adaptation of EMOLL tools in French Simulator User’s guide

12. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Implementation First choice of measures interesting for CAFOC according to its own objectives and priorities. Perspective Client • Measurement of the actual effects of training in a medium term (6 months).What has changed in the work processes and peoples’ behaviour as a result of training. Perimeter of the experimentation : CIRSO – CVRH - Pass Avenir (external clients). • Measurement of learning transfertsWhat knowledge and competences acquired in CAFOC training courses are actually used by trainees in their workplace. Perimeter of the experimentation : GRETA Network (internal client).

13. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Implementation Perspective Finances Measurement of the degree of use of training resources developped or bought by CAFOC ? • Which of these resources are actually used by trainers with their respective trainees • What return on investment Perimeter of the experimentation : GRETA Network.

14. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Implementation Perspective Internal Process Development of training needs analyses In actual practice many training sessions are not based on an actual training needs analysis of the client enterprise or institution but on requests of individual of employees. The objective is to increase the training needs analyses. Perimeter of the experimentation : GRETA Network.

15. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Implementation Perspective Learning and Growth Measurement of the satisfaction of trainees at the end of the trainingThe objective is to refine the present processes and practices Perimeter of the experimentation : all clients

16. Implementation and exploitation of EMOLL methodology at CAFOC Auvergne Implementation - next step 24 May meeting presentation of the HR Scorecard simulator achievement of KPIs sheets

17. WP4 Implementation and exploitation of EMOLL methodology at NEXT MEDIA With the support of the Life Long Learning Programme of the European Union

18. Implementation and exploitation of EMOLL methodology at Next Media Pilot organization : Next Media • A training company • Created in 1988 • 20 employees • With an IT department : • I engineer and 2 IT technicians • This company belongs to AGINIUS corporate network : 10 companies, 40 training centers

19. Implementation and exploitation of EMOLL methodology at Next Media A first training session • May 14 th and 15 th • Presentation of the EMOLL methodology • Project specification • Choice of Kpi’s • Schedule Implementation planning

20. Implementation and exploitation of EMOLL methodology at Next Media Current practices of Next Media in training evaluation • This company already assesses training through a quality process • It is about the satisfaction of the service delivered by the training organisation • Questionnaires are filled by the trainees at the end of the training • These questionnaires are about the training process : content, trainer, organisation, satisfaction and about what was learned • Syntheses are produced based on the answers

21. Implementation and exploitation of EMOLL methodology at Next Media Interest of thiscompagny for the EMOLL project 1 A move from….. to evaluation of satisfaction evaluation of competences 2 key performance indicators of progress 3 automated system of the training evaluation

22. Implementation and exploitation of EMOLL methodology at Next Media The objectives of the project • to evaluate a training programme • by the competences of trainees • from a time T (before training) to another time T+ (after training) • with automated key performance indicators of progress • in order to adapt to companies quality aims and temporal constraints

23. Implementation and exploitation of EMOLL methodology at Next Media The team • TG1 : NextMedia’s training and technical staff • EMOLL partners : • 1 project manager • 1 trainer • 1 specialist of edumetric assessment • 1 statistician

24. Implementation and exploitation of EMOLL methodology at Next Media Tests design • production of 3 pilot tests • test production • use of these tests with a group of trainees • findings and analysis • Production of associated methodology Objective: to be autonomous in this test production • quality of tests -questions, number of questions, grades • item response theory • adequation of test and retest

25. Implementation and exploitation of EMOLL methodology at Next Media Statistics • Objective : • A daily use for 300 to 1000 people a week • Commercial and technical exploitation of the results (learning pathways, training proposals) • Weighting calculations producing indicators few in number taking into account the results of ante and post training evaluations in terms of objectives, expectations, initial levels and reached levels and satisfaction

26. Implementation and exploitation of EMOLL methodology at Next Media Choice of EMOLL Kpis Description of measure: Trainee’s technical competencies are measured before or in the beginning of the training and at the end or after the training. The gap between initial technical competencies and acquired competencies will be an estimation of learning. It will be measured through initial and summative evaluations which are conducted on the same basis. It means that this evaluation has to be done with the same tools.

27. Implementation and exploitation of EMOLL methodology at Next Media Choice of EMOLL KPis Calculation of measure of competences (Final value – initial value) Maximum value × 100 A quantitative and a qualitative indicator!

28. Implementation and exploitation of EMOLL methodology at Next Media Choice of EMOLL KPI’s Description of measure: Trainee’s satisfaction about learning results is measured at the end of the training. It concerns the dimensions of knowledge and of technical competencies acquired during the session. This satisfaction is measured through a likert scale or rating scale were the trainee specify their level of agreement or disagreement on a symmetric agree – disagree scale for a series of statements. Often four or six ordered response levels are used. Ex : I have learned knew knowledge. 1 = strongly disagree ; 2 = disagree ; 3 = agree ; 4 = strongly agree

29. Choice of EMOLL KPis Calculation of satisfaction : Number of respondents giving 1, 2, 3 or 4 Total number of respondents × 100 A quantitative and/or a qualitative indicator?

30. WP4 – Pilot implementation“Progress report” at the end of May 2012 Implementation and exploitation of EMOLL methodology at Next Media • two expert pilot organizations • managers and staff deeply motivated • objectives, results and agendas clearly defined • an experimentation process which may be disseminated • and • STILL MUCH MORE TO DO!

31. THANK YOU for ATTENTION Contact in BPU: Lysiane Lelue – International Relations Office tél +33 473 40 63 37 – lysiane.lelue@univ-bpclermont.fr With the support of the Life Long Learning Programme of the European Union