How to run an R&D survey and setting up and strengthening R&D statistical systems

1. How to run an R&D survey and setting up and strengthening R&D statistical systems South Asian Regional Workshop on Science, Technology and Innovation Statistics Kathmandu, Nepal6-9 December 2010

2. Ch 7 FM - R&D Survey Methodology • Statistics on R&D require regular, systematic and harmonised special surveys • Other sources provide info, but: • concepts of R&D used often different from FM concepts • concepts may change over time • very difficult to obtain all data for the same period • difficult to avoid double counting when tracking flows from financial statements and other sources • Estimates are a necessary supplement to surveys • Especially in higher education sector

3. Scope of R&D surveys • R&D surveys should identify and measure all financial and personnel resources devoted to all R&D activities in all R&D units • R&D surveys are mainly addressed to R&D-performing units • Chapter 7 of the FM only addresses performer-based surveys • Statistical methodologies and other procedures have to be established to capture all R&D, especially for units in the business enterprise sector with little R&D

4. Identifying target population and survey respondents - general • Exhaustive survey not possible in most countries • Constraints include: • number of respondents may have to be restricted to keep costs down • R&D survey may have to be taken in conjunction with another survey • surveys of some groups may require the participation of other agencies with different data needs and hence different questions for respondents • One size does not fit all: every country has different constraints – advice is therefore of general nature

5. Identifying target population and survey respondents - business enterprise sector • The enterprise is recommended as the main statistical unit in the business enterprise sector • Some enterprises perform R&D on a regular basis from year to year, and may have one or several R&D units • Other enterprises perform R&D only occasionally • It is recommended that all enterprises performing R&D, either continuously or occasionally, should be included in R&D surveys.

6. Survey population business enterprise sector – first possible approach • A census-based survey of large enterprises and a sample of smaller ones in order to identify R&D performers and request the information from them • R&D performed in the past in the enterprise is not considered • this is the approach followed in innovation surveys • very small enterprises and enterprises in certain less R&D-intensive industries often excluded for cost reasons • when the sample size is very small, estimates may be less reliable, owing to raising factors • Method not strictly followed in any country

7. Survey population business enterprise sector – second possible approach • Try to survey all enterprises known or assumed to perform R&D, based on a register of R&D-performing enterprises • lists of enterprises receiving government grants and contracts for R&D • lists of enterprises reporting R&D activities in previous R&D surveys, in innovation surveys or other enterprise surveys • directories of R&D laboratories • members of industrial research associations • employers of very highly qualified personnel • lists of enterprises claiming tax deductions for R&D.

8. Survey population business enterprise sector – joint approach Recommendation • To include in R&D surveys of the business enterprise sector all firms known or supposed to perform R&D. • To identify R&D performers not known or supposed to perform R&D by a census/sample of all other firms: • In the industries on the next slide. • In principle, enterprises in all size classes should be included, but if a cut-off point is necessary, it should be at ten employees.

9. Industries to be included Plus any other industry relevant for the country

10. Identifying target population and survey respondents - government sector • Units to include in surveys are: • R&D institutes. • R&D activities of general administrations of central or state government, statistical, meteorological, geological and other public services, museums, hospitals. • R&D activities at the municipality level. • Recommendation: the best way to survey is to send questionnaires to all units known or assumed to perform R&D.

11. Identifying target population and survey respondents - PNP sector • The sources for identifying possible survey respondents are mainly the same as for the government sector. • Register information may be less comprehensive and could be completed by information from researchers or research administrations. • This sector may be more relevant for surveys on R&D funding.

12. Identifying target population and survey respondents – higher education sector • Recommendation: The surveys and estimation procedures should cover all universities and corresponding institutions, especially those awarding degrees at the doctorate level. Other institutions in the sector known or assumed to perform R&D should also be included. • Identification generally easy. • preferable to use smaller units, such as departments or institutes of the university, as statistical units.

13. Working with respondents • Questionnaire: simple and short, logical and with clear definitions and instructions • Optional: simpler survey for smaller units • Test questionnaires on a sample of respondents

14. R&D Manager Better understanding of R&D and FM norms But may not be able to supply exact figures Accountant or personnel manager May not refer exactly to R&D as defined in FM But able to supply exact figures Who is the right respondent? • Cooperation of all three may be needed • Useful to identify in advance the person responsible for providing information and for co-ordinating information from smaller sub-units

15. Encouraging co-operation • Secure co-operation of respondent • Make them appreciate the potential uses of the data • Respect confidential data • Minimise the response burden • Share the results (option: customised information) • Provide technical assistance and contact details

16. Estimations R&D measurement could be done in three stages: • Identification of all specialised R&D units and measurement of their total activity. • Estimates of the non-R&D portions of their activity and subtraction of these estimates from the total. • Estimates of the inputs used for R&D in other units and addition of these estimates to the total.

17. Operational criteria Covered in FM To be covered by data collection agency  keeping good documentation is essential Tools for “translating” theoretical FM concepts into practical questionnaire: • Explanatory notes • Hypothetical examples • Guidance to individual respondents • Documentation on treatment of different cases

18. Estimation procedures • Imputation methods for item non-response • Use previous answer • Hot decking (use info from same survey) • Cold decking (use info from previous survey) • Imputation methods for unit non-response • Use past R&D data (adjusted for sales or employment growth) • Impute as a function of the relation to personnel or sales (test with non-response analysis)

19. Improving Statistical Systems: Advice from the UIS Technical Guide

20. Ch 7 UIS TG: Strategies for setting up S&T statistics systems in developing countries • Institutionalizing S&T statistics • Establishing registers • Structural issues in the private sector and the private not-for-profit sector • User-producer networks • Science & Technology Management Information Systems and other secondary sources • Survey procedures and estimation

21. Institutionalization of S&T statistics • Political support • Infrastructure and sustained staff training/capacity building • Involvement of NSOs: “Official statistics” status for R&D surveys. • Adequate legal framework

22. Establishing registers • R&D in developing countries tends to be very much the purview of public bodies Recommendations: • Establishing a database of public sector R&D projects • include human and financial resources; align with national policies. • design could reflect the R&D statistical reporting/definitions. • source for evaluation of such projects. • Establishing STMIS • provide overview of research system. • framework for establishing complete registers as sample frames for R&D surveys.

23. Establishing registers • Other sources • associations (trade, academic). • learned societies. • registers or databases of scientists and engineers. • database of research grants. • databases of scientific publications. • patents and other IP documents. • business registers.

24. Structural issues in the private sector and the PNP sector • Publicly-owned businesses play a major role in R&D in some developing countries Recommendations: • should consider issuing data for ‘publicly-owned businesses’ separately from the ‘fully private enterprise sector’. • private enterprises could also be disaggregated by ownership, in particular the various degrees of foreign ownership.

25. Structural issues in the private sector and the PNP sector • Business enterprise R&D is presumed to be generally weak in developing countries when compared to industrial countries. Recommendations: • take into account when conducting sample surveys, perhaps by over-sampling, especially amongst larger companies. • big companies should not be missed out as it might imply significant error. • invest time in interviewing key firms to understand their R&D function and obtain a clear picture of their activity. • Private-non-profit sector: make a significant contribution to R&D in developing countries, but the sector tends to be very volatile.

26. User-producer networks Recommendations: • user-producer networks and other forms of stakeholder consultation should be instituted. • establishing national S&T statistics groups. • involve multiple actors. • coordinating/networking among institutions/databases. • partnering with business associations. • conducting face-to-face visits by statisticians and project leaders. • exploit pre-existing personnel ties. • get NSO involved; to deal with privacy of information. • training of interviewers/primary data producers.

27. Science and Technology Management Information System and other secondary sources • STMIS (e.g. database of scientists, research grants, etc): frequent source for the production of R&D statistics. Recommendations: • need close integration between the statistical system and the STMIS. • need adjustments to produce comparable statistics, taking into account issues of definitions and coverage. • need a balanced approach using both STMIS and surveys. • need different approach to Private sector organizations as they are frequently not covered by these systems. • Combined R&D and innovation surveys Recommendations: • the relative rarity of occurrence of R&D in businesses needs to be taken into account.

28. Survey procedure and estimation Recommendations: • attention needs to be paid to questionnaire design. • frequency of survey. • prioritize area of work; accompanied by step-by-step approach. • use of survey questionnaires of other countries for inspiration: need adaptations to local situation. • get expertise from the NSO, in conducting survey, in sampling …. • different questionnaires might be designed for different sectors based on stakeholder consultations. “One size does not fit all”. • procedures need to be developed for estimating missing data.

29. Thank you! http://www.uis.unesco.org m.schaaper@uis.unesco.org