Tensions of Measurement: community investment activities in housing associations

1. Tensions of Measurement: community investment activities in housing associations David Mullins and Vanessa Wilkes, TSRC Housing Studies Association Conference 15th April 2011

2. Summary Sector Context Evidence Base Levels of Measurement 5 levels of measurement Questions and Issues Tensions External Drivers Internal Drivers Conclusions Further Research

3. Sector Context Well established industry, large enterprises (50% market share in social housing) Entrenched hybridity (social & commercial) Core business highly regulated & measured Non-core �community investment� (CI) activities symbol of independence Legitimacy & control issues brought to fore � 2008 Housing & Regeneration Bill & NHF Audit New tools emerging to measure social impact of CI activities (in-house & off the shelf)

4. Evidence Base Scoping study � emerging approaches to measurement of social performance (2010) literature review & checklist (17 tools to set priorities, measure outcomes or link these to corporate goals) 7 Case studies (organisation/tool based: 3 in-house, 4 outsourced) 2 workshops (now 3, plus follow up tracking e.g. of first sector specific tool) NHF Neighbourhood Audits 2008, 2011 Sector overview of CI (64% coverage by stock) Inputs/Outputs (�435mil CI in 6,800 projects 2008; leverage 1.6:2.7) Linked PhD Impacts & outcomes, depth case studies More on this later!

5. The levels of impact measurement OECD definition of impact measurement �the impact of all changes, positive or negative, attributable to the intervention� Also � intended and unintended effects and long-term as well as short-term Original argument (Zappala and Lyons 2009): 3 levels of impact measurement Whole sector Individual organisations Individual projects Tools, frameworks and methodologies not easily transferred between levels Framework developed in relation to housing study (Mullins 2010) to 5 levels Now turning to look at impact measurement and the levels at which it could take place OECD definition of impact evaluation � to this definition should also be added �intentional� and �unintentional� There are a few words in the description which add to the problematic nature of impact measurement � �all� � which includes negative. Given that one of the main drivers for adopting a measurement tool is linked to attracting or maintaining funding, very few organisations / project officers want to report on any negative changes which have occurred. �attributable� � this arguably the most problematic word, how can it be possible to strip out the other influences on the individual or the community? Is there any level at which it is possible to do this? At the individual level, it may be possible to identify the main ones, but this would require an intensive qualitative approach, not replicable at a higher level The �long-term� measurement also contains a number of inherent difficulties given the short term funding of many projects and the change in focus which may occur. The original idea by Zappala and Lyons argues �. We would propose adding another 2 level and contest the fact that there cannot be transfer and interplay between the levels. Now turning to look at impact measurement and the levels at which it could take place OECD definition of impact evaluation � to this definition should also be added �intentional� and �unintentional� There are a few words in the description which add to the problematic nature of impact measurement � �all� � which includes negative. Given that one of the main drivers for adopting a measurement tool is linked to attracting or maintaining funding, very few organisations / project officers want to report on any negative changes which have occurred. �attributable� � this arguably the most problematic word, how can it be possible to strip out the other influences on the individual or the community? Is there any level at which it is possible to do this? At the individual level, it may be possible to identify the main ones, but this would require an intensive qualitative approach, not replicable at a higher level The �long-term� measurement also contains a number of inherent difficulties given the short term funding of many projects and the change in focus which may occur. The original idea by Zappala and Lyons argues �. We would propose adding another 2 level and contest the fact that there cannot be transfer and interplay between the levels.

6. 5 levels of impact measurement Sector level Organisational level  Team level Project level Individual Transferability between levels is dependent on: How the organisation adopts and uses the tool Having the necessary skills and expertise Relationship between internal and external data needs A corporate awareness of what is required from impact measurement The understanding and support by staff Maximising the usage of the data at more than one level We propose that there are 5 levels at which an intervention can be measured. Issues which the both the approach and methodology needs to consider in the right hand column to enable transferability between levels Relationship between internal and external data needs � criticism that using the same tool for this will not lead to effective monitoring. We propose that there are 5 levels at which an intervention can be measured. Issues which the both the approach and methodology needs to consider in the right hand column to enable transferability between levels Relationship between internal and external data needs � criticism that using the same tool for this will not lead to effective monitoring.

7. Questions and Issues Whole sector Is it possible or feasible to measure impact? Dependent on collective approach and understanding Expensive Housing sector � disparate projects and circumstances Draw on information from level 2 Organisation / Team / Project Reason / motivation External requirement? Internal legitimisation / accountability? A learning tool? Availability of skills and expertise Not necessarily new methodologies but alternative application Interpretative and subjective decisions Draw on information from levels below This slide looks at the first 4 levels � not because the individual level is not problematic, but most of the tensions for organisations lie in these 4 areas. Given the expense of measuring impact at level 1 � is it worth it? What will the information be used for? - Dependent on collective approach and understanding across the sector, which may include a common agreed set of indicators such as the value of volunteering or a financial proxy for the money save by preventing homelessness The levels below the macro level are grouped together � in part this is because it may not be possible to have all levels, for instance within a small organisation. Also, some overlap in the issues is apparent. Identifying the reason / motivation for the impact assessment helps to manage expectations. To what extent will it be solely used for external monitoring or is there the opportunity to put it to wider use? Such as a learning tool. Do organisations see the exercise as merely a drain on resources rather than as an opportunity to examine their work in greater detail, the complexity of it, rather than a straight question of what has happened to the beneficiary or community? Will this impact on the methodology used? Does the organisation have the skills to adopt the best and most flexible methodology which would enable data to be used at different levels? This is the level at which there may be a requirement by a funder It may be possible to aggregate data / information / assessments from lower levels Tensions may be highlighted (David to speak more about this) This slide looks at the first 4 levels � not because the individual level is not problematic, but most of the tensions for organisations lie in these 4 areas. Given the expense of measuring impact at level 1 � is it worth it? What will the information be used for? - Dependent on collective approach and understanding across the sector, which may include a common agreed set of indicators such as the value of volunteering or a financial proxy for the money save by preventing homelessness The levels below the macro level are grouped together � in part this is because it may not be possible to have all levels, for instance within a small organisation. Also, some overlap in the issues is apparent. Identifying the reason / motivation for the impact assessment helps to manage expectations. To what extent will it be solely used for external monitoring or is there the opportunity to put it to wider use? Such as a learning tool. Do organisations see the exercise as merely a drain on resources rather than as an opportunity to examine their work in greater detail, the complexity of it, rather than a straight question of what has happened to the beneficiary or community? Will this impact on the methodology used? Does the organisation have the skills to adopt the best and most flexible methodology which would enable data to be used at different levels? This is the level at which there may be a requirement by a funder It may be possible to aggregate data / information / assessments from lower levels Tensions may be highlighted (David to speak more about this)

8. External Tensions External Drivers Funding Sector Promotion Dances around regulation Common outcomes / community budgeting Wider third sector policy influences (e.g. promotion of SROI) Emerging Tensions Data for Funders Conflicts with internal needs Multiple monitoring Timescales Mission Drift Sector promotion �more than a good story� Limited no of PIs obscure complexities of context & impact

9. Internal Tensions Internal Drivers Accountability & Reporting (variety of audiences) Boards & Trustees Management Making sure people enjoy the projects Benchmarking Organisational Power Play Promoting visibility of CI within corporate scorecards Aligning CI activities to corporate goals Emerging Tensions Commercial & Social Goals Social Dividend from commercial success Tendency toward harder measures � leverage Contribute to core business Attribution problems � leads to focus of CI on tenants Corporate v Project Levels Who sets the goals? which CI outcomes get reported?

10. Conclusions Early days but likely to grow High Expectations in advance of evidence! Tensions around types of measures and uses made of the data (both external & internal) Is it worth it? (practical issues, costs motivation/training) Underlying tensions reflect hybrid and large scale of these organisations & competing interests (e.g. Core business v CI)

11. Further Research The role of community investment measurement tools within the organisation How is community investment integrated and understood? What measurement tools are used, what is their strategic fit within the organisation? Data considerations (internal and external) How is data reported and used? What level of understanding does it provide? The practical implications of measurement Does impact measurement cause organisations to rethink their mission, goals or values? What are the operational issues and practical considerations which affect measurement, such as length of contracts My research will focus on why impact measurement is taking place and how the organisation is undertaking it, including the interplay between the measurement tool, community investment and the wider organisation. Further research will take place: - Analysis of the 2011 Neighbourhood Audit and comparison with the previous one Interviews with stakeholders / funders concerning their need for and use of the data. Also the extent to which they guide the development of the data and any specific systems which they use. - A telephone survey of housing associations to identify which measurement tools they are using - Ten case studies with housing associations, to be identified after the Audit in conjunction with the National Housing Federation. A number of personnel will be questioned. I have not yet decided upon specific questions, but these are the areas which I am interested in exploring and the type of questions which seem to be contained within them. These are based on a literature review and discussions with housing personnel Evidence from the Housing Empowerment Network Social impact is not always recognised within organizations which is why it needs to be mainstreamed, rather than be in a community development silo. Tension within the case studies as to what is considered core and non-core and what this means in terms of why social impact measurement is taking place. There is a strong feeling that any measure should also include the fact that a social house has been provided. The NCVO argue for this approach saying that TSOs should measure �full value� rather than added value Does an organisation know in detail what they want from such a tool? For example, the G15 group (group of London Housing Associations) couldn�t agree what they wanted to measure and as many staff are generalists its not always an easy choice. � Issues / gaps in the literature Socially constructed nature of impact assessment makes the task a difficult one. Evaluators are forced to make highly subjective judgements and there is a question as to whether personnel are trained to do this and at what level in the organisation does this take place? There was also a desire for a set of common values such as volunteer time etc from the Housing Empowerment Network to try to help with comparability. There is much room for lack of clarity and confusion between substance and process (Scott, 1992) Data considerations The NCVO argue that one of the key to successful impact reporting is the identification of different audiences and the tailoring of results to them Research by the Charities Evaluation Services also found that despite the increasing demand for impact assessment, funders do not always understand what is realistic for the organisation.My research will focus on why impact measurement is taking place and how the organisation is undertaking it, including the interplay between the measurement tool, community investment and the wider organisation. Further research will take place: - Analysis of the 2011 Neighbourhood Audit and comparison with the previous one Interviews with stakeholders / funders concerning their need for and use of the data. Also the extent to which they guide the development of the data and any specific systems which they use. - A telephone survey of housing associations to identify which measurement tools they are using - Ten case studies with housing associations, to be identified after the Audit in conjunction with the National Housing Federation. A number of personnel will be questioned. I have not yet decided upon specific questions, but these are the areas which I am interested in exploring and the type of questions which seem to be contained within them. These are based on a literature review and discussions with housing personnel Evidence from the Housing Empowerment Network Social impact is not always recognised within organizations which is why it needs to be mainstreamed, rather than be in a community development silo. Tension within the case studies as to what is considered core and non-core and what this means in terms of why social impact measurement is taking place. There is a strong feeling that any measure should also include the fact that a social house has been provided. The NCVO argue for this approach saying that TSOs should measure �full value� rather than added value Does an organisation know in detail what they want from such a tool? For example, the G15 group (group of London Housing Associations) couldn�t agree what they wanted to measure and as many staff are generalists its not always an easy choice. � Issues / gaps in the literature Socially constructed nature of impact assessment makes the task a difficult one. Evaluators are forced to make highly subjective judgements and there is a question as to whether personnel are trained to do this and at what level in the organisation does this take place? There was also a desire for a set of common values such as volunteer time etc from the Housing Empowerment Network to try to help with comparability. There is much room for lack of clarity and confusion between substance and process (Scott, 1992) Data considerations The NCVO argue that one of the key to successful impact reporting is the identification of different audiences and the tailoring of results to them Research by the Charities Evaluation Services also found that despite the increasing demand for impact assessment, funders do not always understand what is realistic for the organisation.

12. Thanks for Listening Please Keep in Touch David Mullins D.W.Mullins@bham.ac.uk Vanessa Wilkes vew930@bham.ac.uk