ICT IMPACT INDICATORS: LINKING DATA FROM DIFFERENT SOURCES

ICT IMPACT INDICATORS:LINKING DATA FROM DIFFERENT SOURCES An EU initiative with 13 National Statistics Offices Tony Clayton UK Office for National Statistics Partnership Event, Geneva Wednesday May 28th 2008

Developing Impact Indicators Agenda • Background to ICT indicator development • Limits to surveys – what we actually measure • Impact assessment; evidence from firms • Using firm level data to develop indicators – the advantages • …. and the challenges to overcome

Background to ICT indicators OECD ‘diffusion’ model, based on linear view • Readiness; ability of firms, individuals to adopt technology • Intensity / use; measures of the proportion of firms, individuals who adopt, and the amount of use • Impact; changes in behaviour, and in economic structure and performance as result of use

Background to EU ICT indicators E- Europe indicators to 2005 focused on use • Citizens • access or ownership of technology • use of internet, and applications • Enterprises • access to or ownership of technology • investment in networks / availability of broadband • use of internet and e-business links • Government • services available on line and degree of take-up • investment in specific areas, e.g. education • Impact? • Only e-commerce, and some individual metrics, to represent changed behaviour resulting from use

Background to EU ICT indicators i2010 more focused on impacts • EU Information space • development of broadband availability, take-up, speed and pricing • multiplatform access in firms • Advanced services • focused on household / individual uses of range of internet services • Security • incidence of problems and solutions • attitudes to risk / trust and confidence • Impact • ICT investment, use and impact on growth / productivity • ICT innovation • ICT R&D and effects on business transformation

Background to ICT indicators ‘Impacts’ now need to reflect innovation in ICT use • Innovation no longer a linear process • depends on networks / feedback / co-creation • ICT as ‘general purpose technology’ • major impacts on economic performance based on use • dependent on range of complementary investments • Innovative uses particularly strong in services • majority of ICT investment in market services • returns to IT investment strongest in services • .. where growth, productivity and innovation are hardest to measure • Pervasive presence of ICT • impacts too important not to be quantified, as part of economic measurement

Limits to surveys Measuring ICT investment is hard to do • Much existing data doesn’t reflect ICT properly as capital • macroeconomic treatment of software investment is inconsistent between countries • firm level source data misses out large parts of ‘own account’ IT software investment – because the respondents can’t quantify it • as ICT service purchases, and IT enabled services, substitute for ‘own investment’, can we pick up the effects? • (Finland results for this study suggests productivity gains from IT outsourcing will ensure this grows) • Measurement differences can be very significant …..

Limits to surveys … and macro data * 2002 Economist Feb 2006

Limits to surveys What else have we learned about indicators? • Surveys about ‘self assessed’ impacts not reliable • assessment of payback from ICT can be inconsistent • links to actual outcomes from production surveys do a better job • Questions on technology less useful than what is done with it • technology changes, and respondents don’t always understand it • effects on behaviour may be a better guide to economic effects • ICT indicators are sensitive to structural differences • industry characteristics can be stronger than national variations • member state estimates may not be comparable, in spite of similar methods.

Who we are • 8 NSIs directly involved in developing research, metadata and method • - UK, Sweden, Finland, Denmark, Italy, France, Austria, Netherlands • Slovenia involved through Ljubljana University • 4 more joined to use method and contribute to analysis • - Germany, Norway, Czech Republic and Ireland • Academic support from Free University of Amsterdam

What we’ve done • Firm level analysis within countries, linking surveys on ICT use, output, productivity, IT investment, skills, innovation … • Core variables from common EU ICT use survey • ‘Add-on’ analysis in countries with extra survey data • Built metadata system to aggregate data for industries / countries consistently • Country / industry indicators built using survey data • Able to bring in other data from National Accounts • Analysis across different environments • New method for indicators

From firm data to macro indicators What we’ve done ….. • Firm level analysis within countries, linking surveys on ICT use, output, etc to identify … • core variables from common EU ICT use survey • ‘add-on’ analysis in countries with extra survey data • Built metadata framework to aggregate this data for industries / countries on consistent basis • country / industry indicators built using survey data • able to bring in other data from National Accounts • analysis across different environments • new method for constructing indicators

From firm data to macro indicators Industry / country aggregates help • We can aggregate to comparable intermediate aggregate data • metadata used to generate completely comparable aggregated data by country / industry / firm size / ownership / market characteristics etc.. • then we can analyse at this level • From ‘industry / country’ analysis we get • ability to do analysis across different national policy environments. • ability to bring in data not available at firm level • quantification of effects at macro level – £billion more persuasive than a regression coefficient!

From firm data to macro indicators Policy Question Research Design Researcher Program Code Publication Metadata Cross-country Tables Network Network members Provision of metadata. Approval of access. Disclosure analysis NSOs

From firm data to macro indicators What NSIs have to do to join ….. • Assemble the data • put production survey, ICT use survey and business register datasets in single research environment • Construct the metadata • describe each survey in standard terms; variable names, format, range, • test linkages between units in surveys using register identifiers • Run analytical code on datasets, using metadata to access and interpret • code uses SAS or STATA operating within NSI • produces results / aggregates which can be shared • disclosure checking before use (can build into code)

ICT metrics From firm data to macro indicators

Contextual variables / complements to IT From firm data to macro indicators

Measures of Impact From firm data to macro indicators

From firm data to macro indicators …and what it can deliver (cont) (example; index of fast-internet enabled workers)

From firm data to macro indicators …and what it can deliver (cont) (example; index of purchases made through the internet)

From firm data to macro indicators …and what it can deliver (cont) (example; index of sales made through the internet)

Impact evidence from firms What microdata contributes to indicators • From micro analysis we get • clear evidence on how inputs to innovation affect behaviour, competition and performance at level where decisions are made • pointers to policy ’levers to pull’ in terms of firm level incentives • limited ability to compare drivers of performnce between countries • Advantages and disadvantages • firm level analysis is flexible, fast, and uses ‘all the data’ in terms of variation / degrees of freedom • but international comparisons of microdata analysis are vulnerable to differences in surveys / definitions / data treatments

Impact evidence from firms Earlier firm level analysis showed ….. • On productivity impact • IT investment and use associated with higher productivity measured through labour productivity or MFP (for a limited number of countries) • Productivity of IT effects stronger in US owned firms, in multinationals, in younger and more innovative firms, and in services • On IT investment and CT deployment • links between communications infrastructure and firm investment IT are significant (for UK at least) • broadband using workers have a significant separate effect on productivity, over and above IT investment

Impact evidence from firms What this study has shown from firm level analysis includes ….. • On productivity impact • consistency across 12 countries in productivity effects of core ICT use metrics for manufacturing firms • much more difference in services; UK NL, France and Scandinavian analyses show significant positive effects, elsewhere mixed picture • On IT investment and CT deployment • broadband is a good indicator – or predictor - of firm level purchased investment • broadband using workers still have a significant separate effect on productivity in countries where use is greatest • ‘joining up workers’ has the biggest effect in differentiated services, where much of the investment is in systems and people

Impact evidence from firms What firm level analysis has shown continued ….. • On Innovation • evidence that connected workers are more likely to innovate • .. From UK that firms which get innovative ideas from third parties are more likely to invest in broadband connections • .. From NL / SE that ICT using firms are more likely to produce innovation outputs, given other inputs, and that productivity effects associated with ICT are strongly influenced by this • => it’s what firms change with ICT that counts, not what they spend • On ICT and skills • higher skills levels increase returns from ICT investment, or from ICT use, whether we use direct measures of skills, or wage measures

Impact evidence from firms What we’ve shown from firm level analysis continued ….. • On organisation • confirmation that some business process links can be associated with significant, positive, productivity effects, in some countries • some of the organisation / maturity variables can be expressed in terms of a composite index … but our surveys need more work • On flexibility / specialisation • Finnish evidence that flexibility associated with mobile access to IT has significant productivity benefits • Finnish evidence that outsourcing of IT services is associated with worthwhile labour productivity advantages

Impact evidence from new indicators What we show from industry / country level analysis includes….. • On ICT use • Evidence that broadband connected employees provide a strong productivity indicator across the contributing countries, over the period 2003-2005 • Looks to be significant across industries, even though firm level effects are uneven • On ICT use as driver of competition • Cross – EU evidence that higher ICT use, and productivity effects is associated with more dynamic competition • Consistent with US conclusion by Brynjolfson et al, that ICT ‘enterprise architecture’ enables firms to replicate successful business models

Country / industry data shows broadband gains:

..and industry split shows differential impacts : • …..industry splits tend to support UK microdata results showing • IT capital matters most in manufacturing • Broadband enabled employees matter more in differentiated services

Advantages Micro based macro indicators can: • Use data available at both firm and industry level • industry indicators compiled ‘bottom up’ from linked surveys • combined with sources from National Accounts or Labour Market • aim to include innovation indicators in next round of work • Build comparisons across countries • firm level data can’t be shared by most NSIs, and are different • metadata approach can iron out differences before analysis • Allow for structural differences between countries • disaggregation possible on completely comparable basis • only need to write analytical code once • It works! • all 13 countries in this project have produced outputs, despite very different systems

Challenges Micro based macro indicators need: • Initial investment in metadata and systems • work to set out dataset and variable descriptions from key surveys • must be able to bring datasets together within each NSI in a common research environment, to enable them to be linked • Systems to control confidentiality within each NSI • research / linking framework must meet national non-disclosure requirements, but still allow firm level data to generate required outputs • Agreement on confidentiality controls for merged dataset • agreement across NSIs on the level of detail which can be used for research and analysis, even if it doesn’t meet national quality standards • agreement on what detail can go into the public domain. • Persistence! • Work in this project produced lots of data which needs simplifying!

ICT IMPACT INDICATORS: LINKING DATA FROM DIFFERENT SOURCES