Analysis of revisions for short-term economic statistics

1. Analysis of revisions for short-term economic statistics OECD Short Term Economic Statistics Working Party Richard McKenzie OECD

2. Analysis of revisions for short-term economic statistics • Main results from OECD revisions analysis study for the index of industrial production • Country comments and reactions • Preliminary results from a similar OECD study on Retail Trade Volume • OECD Main Economic Indicators Original Release Data and Revisions Database • Providing the tools for countries to do their own analysis

3. Revisions analysis for IIP • An attempt to analyse 3 dimensions of statistical quality and make comparisons across countries • Timeliness • Accuracy (assessed through revisions) • Coherency (between IIP and GDP in industry) • Also able to study the interactions between these quality measures • Monthly snapshots of data published in MEI CD-Rom from February 1999 – March 2006

4. Main reasons for revisions in IIP • Late data or revised data from respondents; later correction of errors or imputations • Different estimation methodologies for early estimates and lower sample sizes • Updating of seasonal factors • Benchmarking to other sources (e.g. National Accounts or annual structural surveys) • Changes to base period and changes in statistical methodology, classifications etc.

5. Mean absolute revision to first estimates of year-on-year growth rates

6. Mean absolute revision to first estimates of year-on-year growth rates

7. Are revisions statistically significant? • Mean absolute size of revisions is important • Revisions analysis quantifies this and provides a basis to review our compilation methods • Ideally revisions should centre around zero over time (i.e. equally likely to be + or - ) • If this is not true then reasons causing this tendency in the compilation process must be found

8. Mean revision between first estimates and published one year later

9. Mean revision between first estimates and published one year later

10. Which short term measures should we advise users to focus on? • Are first estimates of month-on-previous-month growth rates from the IIP reliable enough to enable informed decision making? • Not in most countries! First estimates of MoM growth rates revised by 2/3 initial value on average after one year (95% countries revised by more than 2/5) • Shows the value that revisions analysis can provide to users • How to extract the latest signals in estimates without being misleading?

11. Reliability of MoM growth rates to signal expansion or contraction (sign of first estimate compared to that one year later)

12. Do revisions affect identification of turning points? • Very difficult to study • As future turning points interact with past revisions • OECD Composite Leading Indicators dates turning points in IIP using the Bry-Boschan routine • Compared series of first estimates of YoY growth rates with those published two years later, at the BB turning point spots • Where turning points were evident in the YoY growth rates series, preliminary estimates and those 2 years later usually coincided or were very close

13. Timeliness and accuracy • Is there a trade-off between timeliness and size of revisions for the index of industrial production? • Might expect those countries which publish earlier to have higher revisions – at least in the following two months • E.g. due to estimation based on lower response rates, less editing of errors, flash estimation techniques

15. Mean absolute revision to year-on-year growth rates before and after an improvement in timeliness.

16. Coherence analysis • In concept the Index of industrial production should have a strong relationship to Value added in industry from the National Accounts • Coherency test: compare annual growth rates of IIP (most recent series) to annual growth rates in value added in industry from the national accounts • Maybe the countries with high revisions are more coherent (e.g. due to revisions caused by benchmarking?)

17. Correlation between annual growth rates for the IIP and GDP in industry from the annual national accounts

18. Review of country comments to IIP revisions analysis study • Comparisons with similar studies done by the national institute (or comments on results / methods used) • Generally similar results for countries that had done their own studies (some exceptions) • Need to look at other time windows and remove outliers • Sign change analyse of MoM revisions • Impact of seasonal effects should be separated • Few comments on the validity of the timeliness analysis

19. Review of country comments to IIP revisions analysis study • Comments on reasons for revisions and national revisions analysis policies • About 1/3 countries reported aspects of their revisions analysis policy (some with recent changes and efforts to be clear to users in this area) • Need to be wary of revisions analysis as definitive measure of quality (one user emphasised the preference for transparency on statistical methods) • Reasons for revisions covered earlier, all in line with those presented in the paper

20. Review of country comments to IIP revisions analysis study • Comments on the importance of performing revisions analysis for Short term statistics • 2/3 countries stressed the importance of performing revisions analysis • Many countries welcomed the provision of tools and a standardised approach which can save their resources

21. Preliminary results from Retail Trade Volume revisions analysis • Very similar results to analysis of IIP • Mean absolute revisions increase the longer the revision interval • Revision size similar across countries – with some countries noticeably higher and with mean revision statistically significant • First estimate of MoM growth rates subject to large revision within one year

22. Mean absolute revision to first estimates of year-on-year growth rates

23. Mean absolute revision to first estimates of year-on-year growth rates

24. Mean revision between first estimates and published one year later

25. Mean revision between first estimates and published one year later

26. OECD Main Economic Indicators Original Release Data and Revisions Database • Full time series of data published every month starting from the February 1999 edition of the Main Economic Indicators for 21 key economic variables • Access OECD revisions analysis studies for GDP, Index of Industrial Production and Retail Trade Volume • Automated programs and detailed user guide allowing users to perform there own revisions analysis for any country / variable combination available in the database • Information on reasons for revisions and references to international recommendations for establishing revisions analysis policies (IMF SDDS and OECD)

28. http://stats.oecd.org/mei/default.asp?rev=1 Live version

29. Conclusions / Future work • Engaging users on results of revisions analysis studies • Especially the expected reliability of different measures from first estimates – should we do more work to promote more appropriate measures? • How to promote the facilities developed by the OECD • Large saving of resources for NSIs to perform revisions analysis, with OECD or their own data • How to use the results from revisions analysis to improve compilation methods?

30. THE END