October 3, 2013 Lisbon, Portugal EUROMOD Research Workshop

1. October 3, 2013Lisbon, PortugalEUROMOD Research Workshop Testing the Statistical Significance of Microsimulation Results: Often Easier than You Think Tim Goedemé, Karel Van den Bosch, Lina Salanauskaite and Gerlinde Verbist Centrum voorsociaalbeleid Herman Deleeck, Universiteitantwerpen

2. Outline Background and problem statement Statistical discussion Application Last remarks

3. Background

4. A starting point… • A microsimulation study on child benefits and poverty in NMS: • Comparing pre- and post- transfer settings; • Comparing baseline to diverse simulation scenarios; • Poverty rates and confidence intervals reported. • A comment from a colleague regarding statistical significance • The first reaction: … confidence intervals of  estimates should be used for comparing among independent samples or among independent groups. • Relevant for comparing across different household groups; • Not relevant for baseline and reform scenarios – as they are based on the same sample… • Plus … our simulation is a static one … . • Plus… no study really reports such a check (?).

5. An example Where: 1 – total population; 2 – hhs with children; 3 – children; 4 – small families; 5 – large families; 6 – single parent families; 7 – children under age of 6. 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 6 6 6 6 6 7 7 7 7 7

6. Existing literature • In the field of income distribution, tests of statistical significance are increasingly common; • The trend has not yet reached the microsimulation literature;  though attention has been drawn (e.g. Pudney and Sutherland, 1994). • Important to report significantly different (e.g. from baseline) microsimulation results, not the least because of explicitly stated policy lessons, e.g.: • Australia: a 10-percentage point reduction in poverty rate for lone older persons due to simulation of an increase in the age-pension rate. • Simulation of US Social Security reforms: minimum benefits should be seen as an effective poverty reduction tool for low-income earners though poverty rates in baseline and reform scenarios vary by a few percentage points. • A family benefit reform in Lithuania: the limited total poverty effect  might be statistically significant.

7. Why not done? • A lack of attention for estimating statistical inference, could be driven: • By an intuitive notion that sample variation does not play a role, since observed and simulated variables refer to the same sample. • An impression that testing the significance of microsimulation results requires substantial effort, such as employing bootstrapping or some other kind of time-consuming re-sampling technique. • Microsimulations are carried out with programs specially written for this purpose, and thus commands performing significance tests are thus not readily available to microsimulators... • ... . • Often easier than you think!

8. Statistical discussion

9. A standard application • In standard application of sampling theory: • Static simulation of the first-order effects of policy changes; • Simulated variables are calculated using exogenous parameters (e.g. those describing a tax or benefit scheme) and possibly also observed variables (e.g. gross income); • Comparison of observed and a simulated or two simulated variables; • Statistical issues are those of standard survey data analysis - microsimulation does not add any further complications: • A paired t-test  covariation between the two variables; or • A one-sample t-test  calculating the difference between the two variables on the individual level, and applying a t-test on the average of these differences to evaluate whether it is significantly different from zero.

10. Why should we care? • Why should we care about covariance? VAR(D) = VAR(Y-X) = VAR(Y) + VAR(X) – 2*COVAR(Y,X) where, • VAR(D) - samplingvariance of thedifference in the mean (D) of thetwo variable y and y; • Y - mean of variable y; • X – mean of variable x. • If covariance is strongly positive – the usual case in the microsimulation studies VAR(D) ismuchsmallerthanvariance of either of theaverages. • If two samples are independent - covariance is equal to zero.

11. Application using EUROMOD and Lithuanian SILC

12. Settings • 2 microsimulation scenarios, borrowed from Salanauskaite and Verbist (2013): • Abolishing family benefits in Lithuania: distributional impacts of pre- and post-transfer incomes. • Instead of Lithuanian - Estonian family benefits are applied. • Micro-data is mainly derived from Lithuanian SILC: • 12,098 individuals from 4660 households; • A single-stage stratified sample design. • Standard statistical exercise, thought not taken into account: • Simulation errors; • Errors due to the use of uprating factors; • Variance estimates are made using DASP module developed for Stata.

13. Results: mean income, LTL Do reforms result in a significant change in average equivalent disposable income? • In all 3 scenarios: • standard errors are about 24 LTL; • the width of the 95% confidence interval is close to 100 LTL; • confidence intervals considerably overlap. • Diff: (2)-(1): • 26 LTL (< 2% of eq. dpi) increase in mean eq. disposable income is highly significant.

15. Results: GINI Do reforms result in a significant change in GINIs?

16. Results: AROP, % Do reforms result in a significant change in at risk-of-poverty rates? Note: poverty line is subject to sampling variance as the median is estimated on the basis of the sample.

17. Concluding remarks

18. Concluding remarks (1) • Sampling variance cannot be ignored in microsimulation studies working with sample data • When explaining reform effects (i.e. comparing two scenarios), one should take account of the covariance: • which will generally, but not always (e.g. in case of large re-ranking), • result in a high degree of precision regarding reform effects, • even though the sampling variance of the separate point estimates may be substantial.

19. Concluding remarks (2) • Diverse complicated cases do exist: • Budget neutral scenarios: dependence between the baseline and reform scenario; • Reforms in a dynamic model with behavioural effects; • Reforms including some stochastic elements… • Previous research has addressed some issues for these cases, but widely applicable solutions are still to be found.

20. Thank you!