Innovations in governance measurement

1. Innovations in governance measurement Francesca Recanatini & Stephanie E. Trapnell April 26, 2013 Public Accountability Mechanisms (PAM) Initiative

2. Outline • Introductions • Objectives and potential outputs • ISPMS, AGI, and behavioral dimensions of measurement • Public Accountability Mechanisms • De jure and De facto • “Big ideas” for today

3. Objectives of the Workshop • To discuss innovations in the measurement of “de facto” aspects of governance focusing on: • the gaps in existing data coverage, • the strengths and weaknesses of recent data collection methodologies, and • the sustainability of these efforts. • If possible, to identify a “core” set of governance areas and related indicators (both existing and potential)

4. Outputs • Menu of governance indicators and efforts • Draft note on methodologies • Annotated bibliography of sources • Technical expert group (TEG)

5. ISPMS, AGI, and behavioral impacts A proposal and a new way forward

6. The challenge • To measure performance of institutions, and in particular, public sector institutions • To identify areas for institutional reform • To evaluate the impact of change in policies at the country and local level

7. Complementary sets of indicators Indicators of the Strength of Public Management Systems (ISPMS) Actionable Governance Indicators (AGIs) • Measurement of the performance of Public Sector Management (PSM) systems in the middle of the public sector results chain. • Measurement of the way in which the government is held to account through political and non-executive institutions and directly by the public.

8. Public sector results chain within a broader governance environment Broader Governance Environment Broader Governance Environment

9. Moving forward: Utility criteria + feasibility …and 2 “feasibility” criteria.

10. Selected Examples • 1. “Does the executive’s budget or any supporting budget documentation present expenditures for the budget year that are classified by administrative units?” (Open Budget Survey) • 2. Does a legislative committee hold public hearings on the individual budgets of central government administrative units in which testimony from the public is heard? (Open Budget Survey) • 3. Tax payers access to information on tax liabilities and administrative procedures (PEFA)

11. An illustration: Public Accountability Mechanisms De jure vs De facto

12. Public Accountability Mechanisms (PAM)de jure indicators

13. Outputs of de jure data • Library of laws • Analytical publications on the design of public accountability mechanisms • Data, including qualitative and quantitative datasets, country profiles, and descriptive statistics • Country reports on enabling governance environment ….But how do we know what happens in practice?

14. Freedom of information systems in practice:public sector management functions 1.0 Administrative Functions 2.0 Disclosure Functions • 1.1 Facilities • 1.2 Data/Records • 1.3 Human resources • 1.4 Financial • 1.5 Policy • 2.1 Demystification* • 2.2 Responsiveness • 2.3 Appeals • 2.4 Proactive Disclosure * clarity about government processes, rules, and decisions

15. Freedom of information systems:Immediate impacts/Intermediate outcomes • 3.1 Public engagement: Extent to which the public understands, believes in, and engages with the freedom of information process. • 3.2 Government commitment: Extent to which the government supports the freedom of information regime, including efforts to establish participatory decision-making • 3.3 Administrative culture: Extent to which the bureaucratic culture has shifted from principle of secrecy to openness. • 3.4 Operational efficiency: Extent of improvement in operations and decision-making within the organization.

16. How do you measure impacts in a governance “ecosystem” • Participation of civil society organizations and citizens • Accountability and enforcement by parliament, judiciary, police, supreme audit institution and the ombudsman • Political will that enables meaningful change, including protection from retribution, lack of clientilism, and an environment that maintains appropriate incentives for civil servants • Foundational support mechanisms found in public sector systems, such as freedom of information frameworks, conflict of interest rules, financial disclosure practices, and human resources and policy management.

17. Methodologies matter • Surveys: validation issues, response problems • Expert assessment: time-consuming, with degrees of subjectivity • Household surveys: costly, time-consuming, logistical difficulties • Economic/social indicators: collected by whom? National statistics agencies, multi-laterals, NGOs, donors…. • Outcome mapping/process tracing: lengthy, involved, requires local capacity

18. So where do we start?

19. “Big ideas” for today • Actionability and action-worthiness • Change in behavior and not in function • Sustainable measurement practices • Fuzzy concepts and methodologies • Practical applications of data • Challenges in de facto measurement

20. Some emerging trade-offs (or complementarities?) • Global focus versus sub-national focus? • Existing data or new data? • Focused on public sector or broad governance issues? • Perception-based versus objective data? Or both? • Using a single tool or multiple tools? • Detailed focus on a specific case/country

21. Some emerging issues Areas of focus? • Tax administration • Human resources • Rule of law Looking forward • Clarify the focus of the effort and link to a clear research question • Integrate the issue of sustainability • Benchmarking • Understanding strengths of different types of respondents • Clear definition of the context • Dissemination and wide use of the data (and need for a strategy) • New set of tools for certain phenomena (high level corruption)

22. Thank you! PAM website: www.agidata.org/pam AGI data portal: www.agidata.org Metrics & Accountability: http://go.worldbank.org/H1K725TJV0

23. Resources invested in projects to deliver its outputs. Examples: Funding, contracts, materials Desired state of well-being -- a set of conditions, experiences or behaviors – that is the goal for change or improvement. Examples: Maternal/infant mortality rates, Standardized test scores of K-8 students Goods and services produced by the project. Examples: Surveys and Trainings conducted, Laws revised, Agencies established The Missing Middle