Andrey Ivanov , Senior Policy Advisor, Human Development and Roma Inclusion cluster, UNDP BRC

1. Andrey Ivanov, Senior Policy Advisor, Human Development and Roma Inclusion cluster, UNDP BRC Monitoring and evaluation of National strategies for Roma Inclusion: What data and for what purpose?

2. Monitoring and evaluation 101 Basic typology • Monitoring (the process) • Evaluation (of the results) • Intermediary or final • Using indicators • Input Output Outcome Impact Applied at different levels • Of the National strategy • Of the Action Plans • Of Individual interventions Monitoring what determines the kind of data and the kind of indicators used

3. The strategies: How to get results • Having a National Strategy drafted is the beginning, not the end. It needs to be matched by • National Action Plans (usually covering 2 years periods and regularly updated) • Local action plans • Sector specific and integrated projects • In the case of the strategy, for M&E we need • Clear targets– numerical expression of the objectives • Adequate indicators– the definition of the target (how do we measure whether the objective was reached) • Quantitative baseline – the starting point against which the progress/regress is be quantified (the value of an indicator at to) • Milestones – intermediary targets on the way to the general target to keep track of progress (the value of the indicator at t2, t4, t6) • The lower you go, the higher the chances for real inclusion of Roma in the process

4. Different results at different levels • National strategy • Long-term change in the situation of the target group • Difficult to attribute results (but not impossible) • National action plan • Closer link between inputs and outcomes • Clear objectives (that are the strategy’s milestones • Local action plans • Direct link to project outputs • Clear territorial dimensions • Individual interventions • Counterfactual possible although difficult

5. Open questions • What targets for individual priority areas? • Roma specific or general? • What baseline? • 2004? 2011? 2013? • What source of data? • Data availability determines the indicators or the other way around? • What milestones? • The link to individual OPs

6. Example: enrollment rate

7. Another open question: who’s Roma • Politically sensitive (incl. misuse of data for political purposes) • Legal (data protection) or ethical considerations (privacy and fear of stigma) constrains • Insufficient attention to comparability across countries, sub-regions, ethnic groups • The crucial question: what to put in the denominator of an indicator? • The nightmare answer: whatever serves the purpose…

8. Who is Roma? Possible options • Self-identification • Outside (‘imposed’) identification • By non-Roma • By Roma • Combined (multi-stages) – used in the surveys of UNDP (2004 and 2011) and of FRA (2011) • Crucial decision to be made: are we addressing “all Roma” – or “Roma at risk of marginalization”? The answers is both politically and policy loaded.

9. Possible source of data for M&E The data set of Roma vulnerable to marginalization generated from the UNDP/WB regional survey that is part of EU Roma Pilot Project funded by DG REGIO and from FRA Roma Pilot Survey: • Monitoring fundamental changes possible (but not short-term fluctuations). Suitable for National Strategy evaluation • Most indicators have a base-line populated by data from the survey conducted in 2004 by UNDP • The “best game in town” (because it’s the only one…) Caveats: • Still a survey (a sample is always a sample) • Expensive, provides data on “Roma vulnerable to marginalization” – and not on “Roma in general” Other options • Roma boosters in HBS • Longitudinal surveys

10. Survey data is… survey data

11. Going beyond ethnic identity • Be pragmatic - don’t be obsessed by (don’t ask) unanswerable questions like “Who’s Roma?” • But don’t dilute the task of Roma inclusion either • Give priority to socio-economic status • But still keep ethnic identity and specifics in sight • Stick to territorial characteristics • Most of the vulnerable Roma live territorially in separate (segregated) communities • Territorial mapping of those communities is possible • Once a detailed map of Roma-dominated communities is available, it will be possible to correlate ethnic characteristics with territorial tags (individual’s address) • This will allow monitoring a standard set of indicators for a population living in an area with ***% of Roma

12. The benefits of territorial approach • Makes possible to identify the absolute number of the population and not only a percentage • It can be an option solving the problem of individual respondents refusal to declare ethnicity in the census or to declare different one • Less susceptible to political fluctuations • Is more comprehensive in terms of social inclusion (targeting vulnerability per se) • It grasps the marginalized, visibly excluded segment of the Roma population • Actually reflects the fundamental logic of inclusion (including the excluded, not those included already) • Is best for ensuring that control groups (non-Roma living in the same area) are also included

13. An optimal compromise • One approach cannot serve all purposes • Apply different data sources for different planning frameworks • National Strategy – EU-wide survey (representative of… - a matter of political compromise) • National Action Plans – territorially-focused mapping • Individual interventions – project outcome evaluation • Integration of the three levels requires clear milestones in strategies and action plans

14. Conclusions • Integrate the monitoring functions into the entire implementation chain of the strategy • Don’t rely on one source of data and give priority to territorial approaches • Include clear milestones in National Strategies that would serve as a link to the National action plans and OPs • Compete the entire vertical planning and M&E architecture (strategy  plan  call for proposals  interventions) • Go beyond poetry in Operational Programs evaluation building the latter bottom up • Be aware: keeping evaluations vague means keeping them fake