Measuring and Comparing Skills and Skill Levels Not as easy as it looks Professor Ewart Keep ESRC Centre on Skills, K

1. Measuring and Comparing Skills and Skill Levels – Not as easy as it looks?Professor Ewart KeepESRC Centre on Skills, Knowledge &Organisational Performance,University of CardiffKeepEJ@Cardiff.ac.uk

2. YOUR STARTER FOR 10! Q1. What is the lowest number of learning hours associated with a ‘technical certificate’ in a current English ‘apprenticeship’? Q2. What is the pass mark for a ‘good’ (Grade C) pass in GCSE maths? Q3, At Level 2 in NVQ, what is the range of learning hours (between highest and lowest hours normally required)? Q4. What % of your current skills, knowledge, competences are formally certified? CLUE: The world is not as it seems!

3. 3 REASONS FOR ASSESSMENT & MEASUREMENT Formative assessment in order to diagnose weaknesses in ongoing learning, and to guide future progress Summative/terminal certification of what has been achieved at the end of a period/element of learning Comparative systemic measurement

4. THE ACTORS Individuals Employers (a very varied category) The state and policy makers International bodies (EU, OECD)

5. INTERNATIONAL MEASURES In the past, economists had great difficulty in finding datasets that would allow them to measure skill with any degree of accuracy, particularly across countries. They were therefore often thrown back on using proxies for skills, such as: Years of schooling Participation rates for given age cohorts Earnings None of these is a very good. 

6. PROBLEMS WITH THESE MEASURES Years of schooling tell us nothing about the quality of the education or what, if anything, was learned. Ditto participation rates, e.g. Italian higher education. Participation rates may reflect labour market regulation systems rather than skill need, demand or usage patterns. Germany has a very high level of intermediate skills because licence-to-practice regulations mean that many jobs can only be accessed by people with L3. The idea that earnings can be taken as being in more or less direct relationship to skill is a triumph of hope over reality.

7. CURRENT METHODS Qualifications achieved Occupation as a proxy for the skills required. Problem of variations of skill level required within occupations, e.g. ‘manager’. Tests – PISA, IALS, ALLS, etc. Accurate but very expensive. OECD’s 2009 PIAC survey of adult skills, will use tests of ICT, literacy and numeracy on adult workers in different countries. Job requirements – individual responses to surveys, e.g. Skills Survey.

8. THE MAIN CURRENT MEASURE Although there are many different ways to measure skills, public debate and policy in the UK now centres around formalised assessment and certifications systems that generate qualifications (or sometimes parts thereof). In some other countries this is less of an issue

9. QUALIFICATIONS - DIFFERENT ACTORS, DIFFERENT NEEDS Individuals and employers want measurement/ certification as a means of signalling knowledge and skill that is for exchange in the labour market. BUT individuals want certification to cover a broad range of general/transferable learning and to be portable, whereas employers often want it to be narrow, task-specific and non-portable. The state wants standardised output units that can form the KPIs for the public E&T system, and skills stock measures for the national workforce

10. THE BIG PICTURE The performance of a national VET system and skills policy can be conceived of in a number of ways: 1. Internal Performance Efficiency, value for money, unit costs per output Levels of output Levels of participation Inclusiveness Quality of process and output Ability to meet demand (from individuals and employers) 2. The Impact of VET on: Economic Performance Social Performance

11. ONE PERSON’S OUTPUT MEASURE IS ANOTHER PERSON’S INPUT MEASURE People who are interested in VET tend to see the results of education and training as outputs. It is true that skills and qualifications are the outputs of the learning process. HOWEVER, in economic terms, skills are inputs into the productive process (alongside a wide range of other inputs – capital, R&D, etc.). In other words skills only generate an economic output/outcome when they are utilised within the productive process. We will come back to this point 

12. LEVELS OF MEASUREMENT International Region and sub-region Sector and sub-sector (market segment within sector) Particular age cohorts within a population Particular types of skill, e.g. IT graduates Occupational groups

13. COMPARING QUALIFICATIONS STOCKS - THE WORKFORCE AS A WAREHOUSE English policy is driven by a stockpiling approach The labour market is viewed as a vast warehouse, within which boxes are stored – one box for each worker (or potential worker). The boxes are assigned labels denoting the highest qualification held by each worker. Policy tends to view it as an unqualified good to ensure that the greatest number of boxes as possible have as high a number on them as possible. If the proportion of boxes in your national store with high numbers on the side exceeds the proportion in another country, you are deemed the winner!

14. LEITCH – ONWARD AND UPWARD The Leitch Review recommends that by 2020 we try to be in the top 8 at every different level of skill across the OECD (as measured by qualifications in Basic Skills, and at Levels 2,3,and 4. The pile of boxes must grow, and Leitch wants to employers to certify far more of their in-company and informal training

15. CERTIFYING WORK-BASED CPD AND INFORMAL LEARNING Unlikely to grow massively because: Costly Complex Often simply impossible

16. SOME PROBLEMS WITH THE WAREHOUSE APPROACH This approach ignores what is actually in the boxes, or what use it might be put to: Is the content of a Level 3 box in the UK the same as a Level 3 box in Germany? Is one Level 3 box in the UK the same as another in terms of content? Levels tell us nothing about the relevance of the qualification to the labour market. Is a vocational L3 better or worse than a L4 degree in Art History if the job the person occupies is one in retail management? 

17. APPLES AND PEARS? Huge efforts expended on trying to standardise measures at ISCED levels across developed world, but: UK vocational L3 is far narrower in content than a L3 in Germany or Norway. PISA and foreign languages, history, etc. Is a degree from a South Korean university the same as a degree from an Australian university? Same subject/different subject? 

18. QUALIFICATIONS IN ENGLAND – SAME LEVEL, SAME TYPE (VOCATIONAL), VERY DIFFERENT ANIMAL! It is unwise to assume that even in the narrow range of a vocational award at the same level in the same country, that what is inside the box is anything like the same. Construction L3 – 1,220 notional hours IT L3 – 1,475 to 2,400 notional hours Animal Care L3 – 650 notional hours Engineering L3 – 3,900 notional hours Those are official figures. Unofficial industry figures for a L2 in the Fitness industry are 60 hours! Compare with its academic L2 equivalent – 5 GCSEs at A-C grade.

19. QUALIFICATIONS – WHAT DO THEY TELL YOU? Usually they measure (more or less imperfectly) how well someone has acquired and can demonstrate particular knowledge and capabilities. What they are often less good at telling you is: Whether acquisition of the qualification, via a particular course of learning, inflicts costs – for example, by lessening originality or willingness to question basic premises? Whether the individual’s ability to further develop has been enhanced.

20. MORE THINGS QUALIFICATIONS DON’T MEASURE Qualifications only measure some parts of the skill spectrum. For example, they rarely offer much measurement of generic or soft skills. In the service sector, employers are increasingly concerned in the recruitment process to obtain: Generic skills such as problem solving, and inter-personal skills such as communication and empathy. Personal attributes (such as self discipline, loyalty, and punctuality), which may not be skills per se. Aesthetic skills – dress sense, deportment, style, accent, voice. As one UK call centre employer put it, “we recruit attitude”. These skills may be vital, but are usually uncertified and therefore hard to count.

21. OTHER WAYS OF MEASURING PEOPLE-RELATED CAPITAL Business research shows that the private sector’s measurement of human capital often has little to do with the public qualifications system. Measures include: Human capital (vendor qualifications, appraisal, supply chain TQM measures) Intellectual capital Structural capital Customer capital Social capital

22. STRETCHING THE QUALIFICATIONS BOX Employers say that the new Vocational Diplomas (14-19) must cover teamworking. QCA/DfES dream up a written exam in teamworking!

23. DOES SKILL ONLY RESIDE IN THE INDIVIDUAL? Most attempts to measure skill accumulation focus on the skills held by the myriad of individuals in the workforce. However, modern conceptualisations of skill suggest that some skills are held collectively by work teams or communities within organisations. These collective competencies suggest that an organisation’s skills may add up to more than the sum of its individual employees’ skills.

24. MEASURING SUPPLY NOT USAGE International benchmarking exercises (like Leitch) make the vast assumption that if created, additional skills will automatically be used. This may not always be the case.

25. MISSING QUESTIONS OF USAGE Skills deliver enhanced organisational performance within contexts set by the organisation. We know that different forms of: Employee relations context Work organisation context Job design context Will materially impact on how productively enhanced skills can be deployed. Looking at the stock of skills tells us little about how they are being deployed. If the context is poor, skill usage may be inefficient, and productivity benefits from skills smaller than expected.

26. THE UK CONTEXT Chartered Institute of Personnel and Development study in manufacturing: The current managerial approaches to job design are counter-productive….the types of impoverished job we see in manufacturing organisations up and down the country represent waste, on an enormous scale, of the resources, intelligence, skills and energy of those required to perform them. Recent Workplace Employee Relations Survey showed no signs whatsoever that high performance work organisation is spreading across UK economy. Take up has been static since 1998. More skills may therefore be wasted.

27. DECLINING TASK DISCRETION If one element of being skilled is control over how you do your job, the UK workforce is in trouble. The 2nd Skills Survey (2001), showed that: Between 1986 and 2001 there was a 14 per cent decline in the proportion of workers who felt they had a great deal of choice over how they did their work. The fall was sharpest for professional groups (knowledge workers), from 72 per cent in 1986 to just 38 per cent in 2001. Rising skill levels, as proxied by qualifications, have not been accompanied by a rise in job control, quite the reverse. This begs questions about how skills get used.

28. POOR JOB DESIGN, POOR USAGE – ISSUES FOR SKILLS SUPPLY 2001 (UNPUBLISHED) Scottish Adult Literacy research showed that: Reading skills Across the Scottish workforce, the following percentages of workers rarely or never use: Info from computers 34% Letters or memos 26% Bills, invoices, spreadsheets 35% Diagrams 31% Manuals, reference books 53% Reports, articles, magazines 52% Foreign language material 91% 27% of the workforce indicated rarely or never to five or more items. What price boosting adult literacy skills?

29. OVER-QUALIFICATION In Britain, among 20-60 year old employees, the proportions holding qualifications at levels higher than those needed to obtain their current job were: 1986 – 29% 1992 – 33% 2001 – 37% Increasing skill supply and skill stocks may look good in international comparison, but how many of the skills get used?

30. Qualifications Demand and Supply, 2001

31. SKILLS AND WHAT ELSE MAKES THE DIFFERENCE? Research tells us that skills is one among many factors that impact on the performance of firms and the economy as a whole. Unless skills forms part of a balanced scorecard for the economy, we may miss where the real weaknesses in investment lie: R&D Innovation Capital investment in plant and machinery Infrastructure 

32. THE SCOTTISH STORY Over a 20 year period, Scotland has spent about 18% more per head of population on VET than England. Scotland has a larger proportion of its age cohort in higher education than England. The qualifications held by the Scottish workforce are at levels that roughly might be reached in England in about 2020. Is Scotland’s economy performing better than England’s? Is its GDP per head higher? NO. A cause for some heartache among Scottish policy makers.

33. LEITCH – THE BIG ISSUE LEFT HANGING Skills are a derived demand…for the supply of skill to turn from merely potential change in performance into a tangible increase in productivity, the available skills of the workforce have to be effectively utilised. People need to be in jobs that use their skills and capabilities effectively.

34. LESSONS FOR FUTURE RESEARCH Don’t obsess about stocks of qualifications. They can hide poor usage and over-qualification. Remember that qualifications are only a proxy for one part of the skills spectrum. Measures of generic skill formation and usage are very important. Comparisons need to look at demand for skills. The production of masses of graduates, for example, may be a waste of resource if their skills are not actually needed in the labour market. Comparisons need to consider usage of skill. This, in combination with a range of other factors, is important to gauging how well the skills being produced aid economic performance. Supply and demand might be in balance, but if usage is inefficient, or the other drivers of productivity (e.g. R&D) are missing, the economic impact may be small. 

35. MORE LESSONS The non-economic benefits of learning need to be considered much more carefully. The real spill-overs from learning may not be economic, but social. The sustainability and quality of learning provision within national systems is important, not just volume throughputs. Balanced scorecards are vital. Skills alone tell us little.

36. METHODOLOGY IS ADVANCING OECD work EU work – SKOPE/RAND/Danish Technology Institute In the UK Skills Survey 3 will provide an enhanced time series on generic skills, skill usage, and employers’ demand for skills. Many UK agencies are using a wider range of data, in something closer to a balanced scorecard approach. Fixations on qualifications is starting to decline. There is increasing interest in data on usage and demand.