Justifying the Costs of Learning Technology

1. Justifying the Costs of Learning Technology ALT-C Workshop Tuesday 7 September 2010 Caroline Breslin and Diane McDonald BIILS Project University of Strathclyde

2. Workshop Aims • To provide an insight into developing a business case for investment in learning technology via examination of costs and benefits and their evaluation.

3. Format • Overview of BIILS Project and Findings • Measuring Cost • Activity – Identifying Costs • Evaluating Benefits • Activity – Identifying and Evaluating Benefits • Cost Benefit Comparison – Sum Up

4. BIILS Project Background • Benefits of ICT Investment Landscape Study funded in 2008 and 2009/2010 to: • Investigate approaches used to assess performance of ICT projects, services and activities in HE and FE • Provide a synthesis of this range of existing approaches • Develop a categorisation of common terms and strategies • Document the approaches, strategies and metrics used • via • Literature  Survey  Stakeholder Analysis  Case Studies

5. BIILS Project Findings • Most institutions actively carry out ICT investment evaluation... • ... but often not in a structured way and more guidance is required; particularly for the educational context where benefits are difficult to identify and measure. • Note that common practice does not necessarily equal good practice!

6. Business Case • Cost Vs Value • Costs cannot be examined in isolation; value for money can only be assessed in terms of benefits derived from investment

7. PRINCE2 Definition of a Business Case • “The Business Case presents the optimum mix of information used to judge whether the project is (and remains) desirable, viable and achievable, and therefore worth investing in.”

8. Today’s Business Case • Reasons (University Strategy) • Business Options • Expected Benefits (and ‘dis-benefits’ or negative benefits) • Timescales • Costs • Investment Appraisal • Major Risks

9. Learning Technology Investment • Investment in new, open source learning system for students to replace 7 devolved systems which have been implemented in faculties and departments over the last 10 years at Kelvin University. Some of the existing systems are open source, some have been individually purchased and 2 have been developed in-house. Other ad-hoc measures for administrating courses and providing access to learning materials are also in use across the institution. (E.g. Office tools, secured web pages, etc.)

10. Identifying Costs • Include ALL costs which can be attributable to the investment made in order to achieve the potential benefits – wholly attributed or where proportions can be reasonably approximated • EXCEPT those which are small enough to be considered negligible • INCLUDE direct costs, indirect costs and hidden costs

11. Activity 1 – Costs • In groups, identify cost headings for the ‘before’ and ‘after’ scenarios as well as the ‘one off’ costs of change, e.g. ongoing licence/support costs if running old and new systems for parallel or phased implementation. Headings may include • Staff Salary Costs • Training Costs • Capital Expenditure • Revenue Expenditure • Overheads

12. Some Cost Headings Identified by Workshop Groups • Staff • Software licensing • Training • Support • Investigative costs • Time • Student implications • Exit costs – phased migration, etc. • Knock-on costs from audit • Change management costs • Technology costs/transition costs/external consultancy/developers/costs of matching functionality of previous systems • Re-establishing integrations with other systems

13. Notes on Activity 1 – Costs • Activity-based costing • Finance systems (in HE) do not collect figures granular enough to attribute to individual activities (classes, etc.) • Difficult to isolate the contribution of IT within activities like teaching • Full Vs marginal costing • Marginal often used if comparing options – i.e. just look at the difference in costs of options

14. Benefits • A benefit is any positive effect wholly or partially attributable to an investment. • Holistic view of benefits should be taken, possibly rated and weighted. This will be done subjectively but in the context of the institution and its aims. • Evaluation of figures for benefits is therefore a ‘relative’ measure, e.g. ‘before’ Vs ‘after’ investment or comparison of two or more investment options where only one will be funded

15. Activity 2 – Benefits • Identify a range of benefits relating to the new centralised implementation. Weight these to show relative importance and rate them for the ‘before’ and ‘after’ investment scenarios. • Consider organisational benefits (efficiencies, Cost savings, etc.), benefits to core activities (teaching and learning, research, knowledge exchange, etc.) and external/standing benefits (perception, PR, etc.) • Carry out this activity in groups. • Which stakeholder group are you today? • 1. Department with no learning technology systems or expertise • 2. Department with in-house VLE refined over last 5 years by Computer Officer

16. BIILS Project Findings:Types of Evaluation • Financial Evaluation • Target Setting (SMART) • Benchmarking • Informal Review • Investigative • Customer Focused • Compliance • External Accreditation • (9. Combination of two or more of the above)

17. BIILS Framework EVALUATION TYPE Quantitative Financial Targets Benchmarking CONTEXT External Internal Compliance Customer-Focussed External Accreditation Informal Investigative Qualitative

18. BIILS Framework (2) Financial Targets Benchmarking Compliance Customer-Focussed External Accreditation Informal Investigative

19. Reporting on Activity 2 – Benefits • As we go through identified benefits, consider the following: • How will each benefit be assessed in a meaningful way, i.e. how can we determine if and to what extent it has been realised? • Who will conduct this evaluation? • When will it be conducted allowing for the fact that realisation of some benefits may be beyond project timescale?

20. Some Benefits Identified by Workshop Groups • Helping achieve strategic vision of institution Strategy driven (PVC) – measure by target setting and review • Internal PR – measure by informal review • External PR – measure by external standing, e.g. league tables, but how much has investment contributed? – Determine by informal review • Negative benefit – central solution will not meet departmental needs as well as existing devolved solution – benchmarking/comparative analysis to gauge this • Subject discipline needs may not be met by institutional solution (one size does not fit all) • Benefit of non-reliance on single support person which is high risk scenario (vulnerable hardware and staff support set up) • Student experience – more consistency but what about distinguishing provision for particular departments? (I.e. They prefer not to be consistent. – Customer Focussed evaluation is appropriate.

21. Evaluating Benefits - Factors • Type of investment; Infrastructure, Corporate, T&L • Scale of investment • Type of benefit • Scope of benefit • Complexity • Alignment • Availability of comparison data • Type of decision • Institutional culture

22. ReflectionGroup Discussion • Note that example provided was intended to be justified case for investment – real life scenario may be that investment is NOT justified • Continued business justification is required – environment and/or circumstances may change over time and affect viability of project

23. Cost Benefit Analysis Issues • Benefits too narrowly defined • Difficult to isolate specific contribution of particular investment to benefit(s) • Time to realise benefits (could be post-project) • Subjective assessment of benefits where not easily quantifiable • Different stakeholder perspectives – particularly in highly devolved HEIs

24. CBA Issues (2) • Scale of benefits; e.g. How to compare small scale impact but high benefit with wide impact but small benefit • Benefits (and costs) often incremental change rather than radical results • Full realisation of benefits can often depend on parallel initiatives (e.g. New organisational or information management practices)

25. Solutions • Benefits should always be clearly aligned to the stated aims of a project, service or activity. I.e. Evaluation is carried out from the beginning – not afterwards. • Benefits should be aligned to strategic objectives of the organisation • Include all stakeholder viewpoints in evaluation and in benefits impact

26. Reference Material • BIILS: Benefits of ICT Investment Landscape Study • An Evaluation Framework and Toolkit: • http://www.strath.ac.uk/learningservices/innovation/innov_projects/biils/ • Insight Model for CBA in HE: • http://www.strath.ac.uk/learningservices/innovation/projectarchives/insight/ • caroline.breslin@strath.ac.uk • diane.mcdonald@strath.ac.uk