Workshop on Energy Statistics, China

1. Data Quality Assurance and Data Dissemination Workshop on Energy Statistics, China September 2012

2. IRES Chapter 9: deals with Data Quality Assurance and Meta Data Prerequisites of quality – institutional and organizational conditions, including: Legal basis for compilation of data Adequate data-sharing and coordination between partners Assurance of confidentiality and security of data Adequacy of resources – human, financial, technical Efficient management of resources Quality awareness Introduction

3. Under IRES, countries are encouraged to: Develop national quality assurance programs Document these programs Develop measures of data quality Make these available to users Overview of Quality Assurance

4. All planned activities to ensure data produced are adequate for their intended use Includes: standards, practices, measures Allows for: Comparisons with other countries Self-assessment Technical assistance Reviews by international and other users What is a Quality Assurance Framework?

5. Six Dimensions of Data Quality, based on ensuring “fitness for use” Relevance Accuracy Timeliness Accessibility Interpretability Coherence Quality Assurance Framework (Statistics Canada)

6. Should cover all elements of the Quality Assurance Framework Methodology should be well-established, credible Must be easy to interpret and use Should be practical – reasonable, not an over-burden For Key Indicators, see IRES Table 9.2 Quality Measures and Indicators

7. Quality is a priority of senior management Key quality indicators are tracked Quality assurance reviews are conducted for major surveys Data quality secretariat established Questionnaire Design Resource Centre established Quality assurance training delivered Mandatory training provided to new employees Promoting Data Quality at Statistics Canada

8. Quality assurance must be built into all stages of the survey process Survey Stages: Identification of needs Survey design Building the survey Data collection Data processing Analysis Dissemination Archiving evaluation

9. 1. Identification of Needs Activities: Quality Assurance Consult with users and key stakeholders Check sources for quality, comparability Gather input and support from respondents Establish quality targets • Define objectives, uses, users • Identify concepts, variables • Identify data sources and availability

10. 2. Survey Design Activities: Quality Assurance Consult users on outputs Select & test frame Design & test questionnaire Test workflows Develop checklists Develop processes for error detection • Design outputs • Define variables • Design data collection methodology • Determine frame & sampling strategy • Design production processes

11. 3. Building the Survey Activities: Quality Assurance Focus test questionnaire with respondents Test systems for functionality Test workflows Document • Build collection instrument • Build processing system • Design workflows • Finalize production systems

12. 4. Data Collection Activities: Quality Assurance Maintain frame Train collection staff Use technology with built in edits Implement verification procedures Monitor response rates, error rates, follow-up rates, reasons for non-response • Select sample • Set up collection • Run collection • Finalize collection

13. 5. Data Processing Activities: Quality Assurance Monitor edits Implement follow-ups Focus of most important respondents Analyze and correct outliers • Integrate data from all sources • Classify and code data • Review, validate and edit • Impute for missing or problematic data • Derive variables • Calculate weights

14. 6. Data Analysis Activities: Quality Assurance Track all indicators Calculate quality indicators Compare data with previous cycles Do coherence analysis • Transform data to outputs • Validate data • Scrutinize and explain data • Apply disclosure controls • Finalize outputs

15. From IRES Table 9.2, linked to QA Framework Relevance: user feedback on satisfaction, utility of products and data Accuracy: response rate, weighted response rate, number and size of revisions Timeliness: time lag between reference period and release of data Accessibility: number of hits, number of requests Coherence: validation of data from other sources Sample Quality Indicators

16. 7. Data Dissemination Activities: Quality Assurance Format, review, test outputs Produce and follow dissemination checklists Ensure all meta data is available Provide contact names for user support • Load data into output systems • Release products • Link to meta data • Provide user support

17. 8. Archiving Activities: Quality Assurance Periodic testing of processes and systems Ensure meta data is attached • Create rules and procedures for archiving and disposal • Maintain catalogues, formats, systems

18. 9. Evaluation Activities: Quality Assurance Consult with clients about needs, concerns Monitor key quality indicators Periodic data quality reviews Ongoing coherence analysis Investments • Conduct post mortem reviews to assess performance, identify issues

19. Important for assessing “fitness for use” and ensuring interpretability Required at every step of the survey process Critical for enabling comparisons with other data Should include results of data quality reviews IRES table 9.3: generic set of meta data requirements Meta Data

20. IRES Chapter 10 – Dissemination Countries should have a Dissemination policy: Scope of data available Reference period and timetable Data revision policy Dissemination of meta data and quality reports Data collected should not be withheld Users must be aware of the availability Data must be accessible – barriers must be reduced (e.g. format, cost, complexity) Dissemination

21. Individual data must be kept confidential Complicating factors: small numbers of respondents, dominance of a respondent Methods of protecting confidentiality: Aggregation Suppression Other (e.g. rounding) Ensuring Confidentiality

22. An ongoing challenge and trade-off (relevance) Strategies to maximize utility: Raise the level of aggregation Data which are publically available are fully used Request permission to disseminate from respondents Employ passive confidentiality Publish confidentiality rules where data can be disseminated provided “excessive damage” is not caused to the respondent Balancing Confidentiality & Disclosure

23. Users must be aware of availability & release dates Reporting should be based on calendar year (Gregorian) Release targets recommended by IRES: Monthly data within 2 months after reference period Quarterly data within 3 months after reference period Annual data within 15 months after end of reference period Key indicators should be released even faster Ongoing challenge: the trade-off between timeliness, quality Reference Periods and Timetable

24. Countries should develop a revisions policy Provisional data should be revised when new or more accurate data become available Two main types of revisions: Routine revisions (e.g. for late reporters, corrections) Major revisions (e.g. changes in concepts, definitions, classifications, data sources, sample restratification) All meta data should be provided to support users in understanding the revisions Revisions Policy

25. Formats should be chosen to meet user needs Can be a combination of paper or electronic formats Should always include meta data Should minimize barriers to access (e.g. cost, technology, awareness, complexity) Dissemination Formats

26. Statistics Canada – primary data All data are announced in the Statistics Canada Daily Aggregate series available (free) on CANSIM Major publications: Report on Energy Supply and Demand in Canada (Energy Balances) Quarterly Energy Statistics Handbook Move from paper to electronic publications Other major sources of energy information Natural Resources Canada – energy efficiency indicators Environment Canada – greenhouse gas emissions National Energy Board – energy reserves, forecasts, trade Dissemination of Energy Data in Canada

27. Andy Kohut, Director Manufacturing and Energy Division Statistics Canada Section B-8, 11th Floor, Jean Talon Building Ottawa, Ontario Canada K1A 0T6 Telephone: 613-951-5858 E-mail: andy.kohut@statcan.gc.ca www.statcan.gc.ca Thank you!