Mike Grimsley and Anthony Meehan Consultants to EGOV4U WP7 Team Open University, UK

1. EGOV4UE-Government for You: CIP ICT PSP Pilot B Grant 250509Work Package 7: Impact Evaluation of Multi-channel eGovernment services tackling disadvantage and social exclusion in the cities of Milton Keynes, Dublin, Reykjavik, Rijeka and the Island of Malta. Mike Grimsley and Anthony Meehan Consultants to EGOV4U WP7 Team Open University, UK Paper presented to Regional Studies Association Winter Conference Contested Regions: Territorial Politics and Policy London, 25th November 2011 Grimsley and Meehan RSA (EGOV4U) Nov. 2011

2. Outline • Beyond efficiency • Muti-channel context: the scope of impact evaluation • Forms of evaluation and governance • EGOV4U Impact Evaluation Framework Grimsley and Meehan RSA (EGOV4U) Nov. 2011

3. Problem of Evaluation • “Whilst member states share a perception of the benefits of digital inclusion, they [have] difficulties in substantiating those perceptions.” [1] • “[Benefits] cannot be defined with confidence in a similar way [to costs] as they are too qualitative, intangible or unpredictably set in the future.” [2] • “The social impact of ICT is still a basically unexplored area at the European level.” [3] • “Statistics on social impacts of ICT tend to be of an intermediate nature, for instance, impact on patterns of work or changes in how people do their shopping, rather than whether this has a positive or negative outcome for individuals, communities or the broader society”. [4] • EGOV4U addresses this problem by advancing a generic, high-level, Impact Evaluation Framework (IEF) applicable to eGovernment/eInclusion initiatives addressing disadvantage and social exclusion. Grimsley and Meehan RSA (EGOV4U) Nov. 2011

4. 3rd sector providers Scope of Multi-channel Impact Evaluation political, social and economic institutions private providers social enterprises community – family, friends, neighbours Grimsley and Meehan RSA (EGOV4U) Nov. 2011

5. How do different approaches to evaluation shape the character of governance? Evaluation: macro, meso, micro, mixed? Metric-based enquiry to improve local performance Metric-based league tables to rank or compare EGOV4U Longitudinal Study: Phase 1 focus on individual initiatives Phase 2 learns lessons by comparing and contrasting initiatives Contextual enquiry / Action Research to shape and learn from local practice Comparative enquiry for learning and theory building (collaborative benchmarking) Grimsley and Meehan RSA (EGOV4U) Nov. 2011

6. EGOV4U Impact Evaluation Framework • In the context of Multi-channel eGovernment Services… Evaluate what matters for economically, socially and politically sustainable communities: • Outcomes and (transformative) impacts (not just inputs or outputs) • ‘Community Capitals’ as high-level measures capturing: • what people want for themselves and others – Collective Well-being or Public Value (Moore, 1995) • transparent governance for sustainable eServices Grimsley and Meehan RSA (EGOV4U) Nov. 2011

7. Why ‘Community Capitals’ • ‘Community Capitals’ : • are capable of ready and meaningful interpretation in diverse contexts • link impact on individuals to impact at a societal level • support comparison and learning between local projects, but also some degree of comparison of projects at national (meso) or international (macro) level • are directly relevant to high-level policy goals aiming to address social exclusion and disadvantage through digital inclusion and e-service projects. • relate to known barriers or obstacles to effective longer-term impact of eGovernment and other e-services • are widely recognised as being positively causally correlated with community and societal well-being and sustainability. Grimsley and Meehan RSA (EGOV4U) Nov. 2011

8. Community Resources/Capitals • Capitals: • Human: collective knowledge and skills (technology specific and technology independent); • Social: (bonding, bridging, linking); • Organisational: processes; managerial and governance structures of community; • Environmental: digital and non-digital amenities that facilitate co-production of outcomes and impacts; • Infrastructural: ICT and related infrastructure, including ‘back-office’ systems; • Financial: individual and collective financial resources; • Reputational: Trust and recommendation (underpinned by transparency, security, privacy); democratic endorsement. • Taken together, Community Capitals constitute a ‘Habitus’ that bounds the attainment of sustainability and cohesiveness for a community. Effective policy interventions are those that ‘enlarge’ the Habitus. Financial capital Infrastructural capital Human capital HABITUS Reputational capital Organisational capital Social capital Environmental capital Grimsley and Meehan RSA (EGOV4U) Nov. 2011

9. e-Service (Co-) Production Context Reputational Capital Human Capital Financial Resources Environmental Capital Organisational Capital Social Capital Infrastructural Capital Project Impact and Governance MC-Network Beneficiaries Reach and Diversity (Inclusivity) Transformational Impacts (Effectiveness) Public Value Transactional Impacts (Efficiencies) Multi-channel e-Service Network(s) Governance Context Needs, Entitlements, Policy Priorities Grimsley and Meehan RSA (EGOV4U) Nov. 2011

10. Impact Evaluation Levels • Level 1: Achieving (quantitatively expressed) eService objectives • Level 2: Impact on stakeholders in the service delivery network • 2a: beneficiaries/clients and close intermediaries (e.g. family, friends, carers) • 2b: organisations • Level 3: Wider social and public (“democratic”) value Grimsley and Meehan RSA (EGOV4U) Nov. 2011

11. L2a: Beneficiaries & Intermediaries • Clients, family, friends, carers, and others who are not part of formal intermediary organisations, • Indicative measures of service users’ experiences and outcomes: • availability, usability and accessibility of service channels (ICT-mediated, face-to-face, one-stop-shop, family, friends, carers or neighbours); • skills enhancement; • health (anxiety/stress) • quality of relationships between users, their intermediaries, service personnel, organisations providing the services, and the political institutions commissioning them – key measures are a sense of trust and a willingness to recommend the service(s) to others; • sense of ‘empowerment’ in terms of three key correlates (factors): a sense of being well-informed, a sense of personal control in their lives and a sense of influence (in respect of the service and more widely) Grimsley and Meehan RSA (EGOV4U) Nov. 2011

12. L2b: Organisations • Local government, public sector agencies, commercial organisations, 3rd Sector, social enterprises, beneficiaries and their representatives • Sustainability of MC eService Networks addressing social exclusion’: • evolution of multi-channel service delivery (especially ability to engage intermediaries and beneficiaries) • improvement in the identification of (holistic) personal needs • enhancements in methods of assessing and measuring outcomes/impacts • relevant changes in organisation structures, relationships, policies and systems designed to support meeting outcomes/impacts Grimsley and Meehan RSA (EGOV4U) Nov. 2011

13. L3: Social and Public Value Legitimacy of services and service providers (inc. government): • perceptions of efficiency and effectiveness • democratic (public involved in determining policy for service provision) • transparency (access to services is based upon open and observable criteria) • equity (citizens treated fairly - on the basis of need as opposed to age, ethnicity, health, mental wellbeing, etc.) • citizens trust service providers Outcomes beyond those delivered by specific services: • enhanced quality of life, wellbeing and happiness • increased social cohesion, social inclusion, social capital • safety and security - both objective and subjective (‘ontological security’) • equality, for example reducing deprivation and social exclusion • promoting democratic and civic engagement Grimsley and Meehan RSA (EGOV4U) Nov. 2011

14. References • i2010 eGovernment Action Plan Progress Study: Final Report. Prepared for the European Commission by: The Danish Technological Institute, Denmark, and TNO Information and Communication Technology, The Netherlands. European Commission, November 2009. • Breaking Barriers to eGovernment: Overcoming obstacles to improving European public services. Oxford Internet Institute, eGovernment Unit, DG Information Society and Media European Commission (2007), p6. • Benchmarking Digital Europe 2011-2015 a conceptual framework. i2010 High Level Group, European Commission 2009, p15. • Measuring the impacts of ICT using official statistics. Working Party on Indicators for the Information Society, OECD, October 2007 • Compendium of OECD Well-being Indicators, OECD, 2011 • Eurostat (2011) Sigma - The Bulletin of European Statistics, 2010/02 Grimsley and Meehan RSA (EGOV4U) Nov. 2011

15. Supplementary slides for use in discussion Grimsley and Meehan RSA (EGOV4U) Nov. 2011

16. eGOVERNANCE STRUCTURES & PROCESSES Economies of Effectiveness and Efficiency In pursuing Equity, Effectiveness and Efficiency originate in different economies: ‘transformational’ and ‘transactional’. In the context of multi-channel eGovernment/eInclusion services addressing disadvantage and social exclusion: CREATING PUBLIC VALUE-ADDED eGovernment Services EGOV4U TRANSFORMATIONAL ECONOMIES EGOV4U TRANSACTIONAL ECONOMIES IMPACT EVALUATION FRAMEWORK (IEF) EGOV4U FIELD OF TRANSACTIONS • Transaction economies are associated with aligning internal operational decisions of multi-channel eService providers with service transactions. These economies relate to current or backward-looking perspectives of the organisations they occur in. They are set in relation to the costs of administering or running a provider organisation and are thus closely related to transactions costs. • Transformation economies are associated with strategic decisions over the outputs of the multi-channel eService and responding to and influencing the (democratic) governance and regulatory environment. They tend to be associated with forward-looking perspectives and are set in relation to external goals and transforming the performance and future of the community or society GENERATING SOCIAL INCLUSION Malta Grimsley and Meehan RSA (EGOV4U) Nov. 2011

17. Well-being (OECD) OECD (2011) Compendium of OECD Well-being Indicators, OECD Grimsley and Meehan RSA (EGOV4U) Nov. 2011

18. Well-being: what people want (OECD) Martine Durand, OECD Chief Statistician, The OECD ‘Better Life Initiative’, Measuring National Well-being, London, July 2011. Grimsley and Meehan RSA (EGOV4U) Nov. 2011

19. ONS(UK) National debate 2010-2011: 175 events with 7000 attendees; 34000 online responses. What people want (ONS) Jil Matheson, National Statistician (UK), Measuring National Well-being, London, July 2011. Grimsley and Meehan RSA (EGOV4U) Nov. 2011

20. EC (Eurostat) • European Commission Communication ‘GDP and Beyond’: • Stiglitz-Sen-Fitoussi report on measuring social progress overlaps with the needs of the Europe 2020 strategy for smart, sustainable and inclusive growth. • Key dimensions: material living standards; health; education; personal activities – including work; political voice and governance; social connections and relationships, environment, as well as physical and economic security. Eurostat 2011 Grimsley and Meehan RSA (EGOV4U) Nov. 2011