A tool for a snapshot assessment of a national statistical system

1. A tool for a snapshotassessment of a national statistical system Quality conference – 3-5 June 2014 Vienna Claudia Junker, Head of Unit "Statistical Cooperation", Eurostat

2. Content of the Presentation • Background • Objectives • Presentation of the tool • Filling in the tool • Using the tool • Conclusions

3. Background • Original context is development cooperation, but the tool can be used also in a wider context • Results-based management and performance assessments are central in development • The monitoring of development policies and of aid delivery (and Budget support in particular) require the setting of benchmarks and the construction of systems of indicators • These benchmarks and systems of indicators are fed with public statistics which are provided by the countries themselves • Countries need baseline and benchmarks for assistance projects and their own development => What do we know about the quality of these statistics and how can we assess it in a user-friendly way?

4. Objectives: To design a user-friendly tool to help different stakeholders to assess: • the quality of the data donors receive from the partner countries to monitor the results achieved by the national policies and reform programmes, • the performance of the support given by the EC to these policies and reform programmes, • The success of assistance projects visible in corresponding official statistical data • The state-of-play in the development of the NSS in a country, • The efforts to improve the performance of the NSI/NSS => A natural vocation for supporting quality

5. Eurostat's commitment to quality statistics • European Statistics Code of Practice • Regular self-assessments by the EU member states • Peer reviews and Global assessments with partner countries (East and South) • Support to the African Charter on Statistics and similar regional initiatives => A natural vocation for supporting quality

6. Presentation of the tool The tool consists of two modules: • Module 1: analyse and document the functioning of the national statistical system (NSS) as a whole, with a special focus on institutional, legal and organisational aspects. • Module 2: analyse and document the statistics in specific sectors. Awide range of sectors can be addressed (such as: education, health, agriculture, food security, private sector development, climate change and environment, economic growth, governance, infrastructure, external trade of goods, employment, living conditions, macro-economic stability, population).

7. Presentation of the tool Module 1: Overall National Statistical System (56 questions)

8. Presentation of the tool Module 1: National Statistical System (56 questions) Examples of questions: • Are statistics used in the development, implementation and monitoring of national development policies? • Are there statements made by high-level officials (president, ministers, parliament, etc.) on statistics development? • Is statistical independence a guiding principle for the functioning of the NSS? • Does the NSS have mechanisms for assigning of responsibilities in the national statistical process? • Is there a human resources policy (recruitment, career development, education and training) for the NSI? • Which proportion of the staff has a (permanent) internet connection? • Are there internal handbooks / guidelines / recommendations for the statistical production processes? • Are there formal processes in place to consult users about their statistical needs?

9. Presentation of the tool Sector matrix

10. Presentation of the tool Module 2: Sector statistics (between 59 and 61 questions)

11. Presentation of the tool Module 2: Sector statistics Examples of questions: • Is the development of sector statistics included in the Sector Development Strategies? • Which proportion of the staff involved in producing sector statistics is equipped with a modern computer? • Is information on the methods and procedures used for the sector statistics published on the NSI or Ministry website? • Do all users get access to statistical releases at the same time? • Are there partnerships or joint studies/works with the scientific community? • Are their quality controls and checks on the reporting made by the concerned administrative units (schools, health centres …)? • How much of the statistical outputs in the sector are available via the internet? • Is there a contact point to assist users?

12. Presentation of the tool Easy colour coding: Far from complying with quality requirements Not totally complying with quality requirements Complying with quality requirements Objective criteria – Column in the matrix with management comments for filling in the tables

13. Presentation of the tool Additional coding: Recent progress No changes Deterioration More subjective elements

14. Presentation of the tool

15. Presentation of the tool

16. Presentation of the tool

17. Filling in the tool What are the basic sources? Aspects linked to the legal and policy framework, to the organization and functioning of the system For Module 1 (mainly themes 1 and 2) For Module 2 (mainly theme 1)

18. Filling in the tool What are the basic sources? Aspects linked to practices: complements to the basic sources For all modules (mainly themes 3, 4 and 5 for Module 1 and themes 2, 3 and 4 for Module 2)

19. Filling in the tool Who will do it?

20. Using the tool Results Module 1: example

21. Using the tool Gap in resources allocated to statistics Results Module 1: example No rigor in quality controls Strategy exists but not fully mainstreamed in policies Poor relevance of and accessibility to data Legal framework in place but poor coordination

22. Using the tool Results Module 1: example

23. A snapshot of an NSS – using the tool Results Module 1: example Limited information available on financing Lack of trained and experienced personnel Sectors with no handbooks and no systematic quality controls Share of responsibilities and coordination is established but poor execution in practice Number of PC ok but poor internet access No rules for the correction of serious errors The law is very comprehensive but the notion of confidentiality is not yet fully internalised No release calendar No cooperation with academics Participatory process for the design of the strategy but not translated into wok plans No systematic consultation with the users Poor usability of the web site for data dissemination GDDS not joined, only a few statements by high-level officials on statistics

24. Using the tool Results Module 2: specific indicators

25. Using the tool Results Module 2: specific indicators Only one producer and availability on time Definitions and sources not in line with international recommendations Only short term series and data not easy to find for users

26. Using the tool An instrument of statistical quality assessment for the EU Delegations • The tables could be filled and updated regularly as a monitoring instrument for the EU Delegations to assess the development of the statistical system ofpartner countries

27. Using the tool An instrument of dialogue on statistics between donors (EU or others) and the partner countries • The tool could be embedded in annual reviews or monitoring exercises that could be done in the framework of a political dialogue between the donors and the country authorities, • The tool could ease identifying statistical areas in need to be supported by the donor community and help defining assistance programmes

28. Using the tool An instrument of assessment of the progress made in the NSS • Filling the tables could be the result of a coordination work within the NSS. The results could be presented and discussed each year (regularly) at a meeting of the National Statistical Council or a similar entity. This would help adjusting work plans. • Examples and good practices could be drawn from exercises carried out in other countries.

29. Conclusions What the Snapshot is: In photography: a photograph that is "shot" spontaneously and quickly. The tool is a complex camera (comprehensive and articulated framework) that allows shooting spontaneously (the tool is available and can be used/understood by non-statisticians) and quickly (the tables can be filled in rapidly with available information) In computing: the capture of an object's state at a specific moment in time The tool gives a picture of the statistical system of a country (or of sector statistics in a country) at the moment when the related tables are filled in. This picture can be taken at regular intervals (e.g. yearly) in order to monitor the progress made

30. Conclusions What the Snapshot is not: An instrument for rating and/or comparing countries • The tool is “process-oriented” and aims at supporting a dialogue among the various actors involved in the development of statistics in a country A detailed analysis of the sector statistics • There are more sophisticated and targeted frameworks for some statistical sectors (such as Education/UNESCO) that detail the statistical processes leading to data production in more depth. The tool can be fed from these in-depth analyses and can also be an initial step before going deeper into the analysis.

31. Conclusions • Freely available from Eurostat upon request to send at:ESTAT-STATISTICAL-COOPERATION@ec.europa.eu • Available in three languages: French, English and Spanish • Manual and a user-guide • E-learning training course on how/why to use it