Criteria and methodology for the evaluation of minority policies

1. Criteria and methodology for the evaluation of minority policies Informal International Consultative Meeting in the Area of Minority Issues, European Centre for Minority Issues, Flensburg, 16-18 September 2004 François Grin University of Geneva & Education Research Unit

2. Structure of presentation • Introduction: rationales for action • Three features of policy analysis • Applying policy analysis: six « vignettes » • Conclusion: implications for minority policy development

3. Combining different rationales for action in minority policy • Principle-based:  application of norms and standards, e.g. rights-based approach • Goal-oriented:  selection and design of policies yielding the « best » results.

4. Goal-oriented: types I and II • Type I: goal-oriented approaches give substance to principles (principles precede analytically-informed action) • Type II: Goal-oriented approaches shape principles (analysis precedes the setting of standards)

5. Different subcultures: the diversity clover • It is important to remember that different subcultures exist in the area of minority issues. • Four main families can be identified among professionals as well academics, centred respectively on the issues of « national minorities », « language minorities », « migrants » and « indigenous peoples ». • Interaction between these « families » and their respective « subcultures » remains limited, but needs to be developed in order to locate the most appropriate expertise and intellectual resources.

6. Policy v. political questions • What (exactly) should we do (once principles are by and large agreed upon)? • How should we do it?  Choice and application of criteria  Contribution of policy analysis in three principles and six « vignettes »

7. Basics of policy analysis • Aims at creating knowledge about the consequences and performance of possible (ex ante) or existing (ex post) policies • Stresses the cause-and-effect links between policy action (upstream) and results (downstream) • Relies on the principle of comparison between competing options

8. Methodological implications Policy analysis therefore requires: • Careful investigation of the cause-and-effect relationships through which policy decisions are related to results  model building • Identification and measurement of policy measures (« policy inputs »), intervening variables, direct policy effects (« outputs ») and results in terms of the processes actually targeted (« outcomes »).

9. Vignette #1: outcomes v. outputs • Outputs: direct effect of a policy • Outcome: final effects of a policy, in terms of the variables one wishes to influence. •  Policy measures produce outputs, which influence processes that result in outcomes

10. Vignette #2: on the nature and role of analytical models • A model is needed to connect the policy decision « upstream » (A) to its direct « outputs » (B) through to the resulting «  outcomes » (C) that occur downstream, given certain intervening variables (D). • B=f(A) and C=g(B, D), hence C=g[f(A), D] • Rather than a representation of reality, a model is a tool to help us think about reality  focus on causal links, with particular attention to necessary and sufficient conditions • A model is « a metaphor whose implications have been spelled out ».

11. Vignette #3: dearth of targeted models • Relatively little is known about the causal relationships between policy measures « upstream » and outcomes « downstream », when the outcome is (i) a defining feature of minorities, or (ii) a variable whose level is structurally correlated with minority identity or experience. • Need for model development  example: the P-TOP (« policy-to-outcome path »).

13. Vignette #4: working backwards • Identify the desired outcome (e.g.: minority language use) • Then analyse this outcome as the result of a process in which different variables operate • Then select, among these variables, those that can be influenced through policy measures.

14. Vignette #5: The role of indicators Indicator: measurement of a variable which is relevant to the policy-influenced causal relationship. Indicators must be: • Context-specific • Non-circular • Clearly located along a policy-to-outcome path ( (i) « systemic » (ii) « interrelated » (iii) [some] reponsive to policy) • Relatively easy to collect

15. Vignette # 6: criteria for good/best practice/policy

16. Defining and understanding criteria • Resource allocation (1): effectiveness (« making a difference ») • Resource allocation (2): cost-effectiveness (« least-cost effect ») • Resource distribution: fairness as result; democracy as process • All four criteria can be measured through indicators

17. Implications [1]—we need to: • … distinguish clearly between the outputs and the outcomes that can be associated with each option. • … develop a model linking the policy measure, its outputs, and the outcome; identify causal links, necessary and sufficient conditions, etc.

18. Implications [2]… and to: • … explicitly state the substance of « effective », « cost-effective », « fair » and « democratic » in the context of the policy/programme/project under consideration • … pick or design indicators for effectiveness, cost-effectiveness, fairness, and democracy, making sure that these indicators display the necessary properties