1 / 37

G. S. Kearns Information Resources Management Journal Vol. 17, No. 1, pp. 37-62 Jan-Mar 2004

A Multi-Objective, Multi-Criteria Approach for Evaluating IT Investments: Results from Two Case Studies. G. S. Kearns Information Resources Management Journal Vol. 17, No. 1, pp. 37-62 Jan-Mar 2004. Outline. Introduction The IT Investment Decision The Analytic Hierarchy Process

fausta
Download Presentation

G. S. Kearns Information Resources Management Journal Vol. 17, No. 1, pp. 37-62 Jan-Mar 2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A Multi-Objective, Multi-Criteria Approach for Evaluating IT Investments: Results from Two Case Studies G. S. Kearns Information Resources Management Journal Vol. 17, No. 1, pp. 37-62 Jan-Mar 2004

  2. Outline • Introduction • The IT Investment Decision • The Analytic Hierarchy Process • The IT Investment Model • An Information Systems Example • Evidence From Two Case Studies • Results of Investment Decisions • Discussion • Study Contributions • Conclusions

  3. Introduction (1/3) • A majority of CEOs • IT investments were economically infeasible • Confidence about the future ability of IT to provide strategic advantages • Economic analysis of IT returns relies on quantitative measures

  4. Introduction (2/3) • Traditional approach • Have not proven useful in the economic evaluation of IT-based investments • Single criteria techniques • Discounted cash flow, cost/benefit analysis • Bias towards the tangible benefits • IRR or net present value may ignore the ‘soft’ , qualitative benefits of IT applications • Strategic applications • Require a method • Reliably measure all benefits in a consistent manner that is understood and supported by management

  5. Introduction (3/3) • Maximizing returns from IT investments requires a total portfolio planning approach • Can not be accomplished by valuing each investment individually • Mutually exclusive, mutual dependencies • Should not be combined due to the total risk • Combined with integer programming and the Analytic Hierarchy Process • Support a multi-objective, multi-criteria approach • Address several issues hindering the success of IT investments • The purpose of the paper • Demonstrate the MOMC approach to IT investment analysis • The applicability of the proposed model using an illustrative example of five information systems projects • The results of two case studies in which the model was successfully applied

  6. The IT Investment Decision (1/3) • There is little persuasive evidence • Investment in IT positively impacts the financial position of the firm or increases productivity • Measurement problem • Time period between investment and realized benefits • The direction of causality is difficult to prove • The study examines a more direct method of influencing business performance • Improving the quality of the IT investment portfolio

  7. The IT Investment Decision (2/3) • Traditional financial accounting measures • Past evaluation of IT investments suffer from • Isolation • Difficulty in valuation of benefits • Low explanatory power • Ignore basic investment tenets • All financial measures are sensitive to the valuation of benefits • The approach assumes that each investment stands on its own merits without regard to other investments • Some investments generally have failing marks under ROI and passing marks under net presents value • Such as ERP

  8. The IT Investment Decision (3/3) • IT-related investments represent in excess of half the annual capital expenditures for many firms • An agreed-upon approach to measuring IT investments does not exit • Returns on IT investments have been unsatisfactory • The selection of IT-based investments • Produce the highest value for the firm • Value must reflect a combination of both quantitative and qualitative criteria • A decision support process is needed that will incorporate all relevant decision criteria

  9. The Analytic Hierarchy Process • AHP applications are numerous • Strategic planning, microcomputer selection, etc • AHP combines with other techniques • multi-dimensional scaling and integer and linear programming • No prior illustrations of this use • The MOMC is an effective measurement process • Rank alternative investments according to criteria • Corporate strategies • The strict time constraints of the planning process • Support consensus among a diverse group of individuals • Reflect investment precedence or exclusivity constraints • Incorporate both quantitative and qualitative criteria • Be understood by management

  10. The IT Investment Model (1/7) • Corporate strategies used as project ranking criteria • The importance of linking IT strategies to corporate strategies has been well known • Traditional discounted cash flow techniques lack linkage to corporate strategy • AHP facilitate specification of criteria based upon corporate strategies

  11. The IT Investment Model (2/7) • Level of difficulty • Include • The flexibility of the measurement process in reflecting changes • Perform sensitivity analysis • Produce viable alternative solutions • Provide an explanatory trail • AHP methodology • Use a paired-comparisons approach • The criteria indicators represent typical investment alternatives • The sum of each criterion’s value becomes the investment’s global score for final ranking

  12. The IT Investment Model (3/7) • Explanatory power • The most valuable feature of AHP • A convenient framework for concise representation • Offer a formal, systematic, consistent approach • When combined with an integer optimizing model • The weights can readily be compared • Managers are able to see into the process

  13. The IT Investment Model (4/7) • Creating consensus • AHP is highly effective in distilling information from groups and fostering consensus • By paired comparisons • An important foundation for acceptance • AHP creates quantitative rankings • Use a systematic approach to capture priorities • Measures the consistency of the overall process • Cost, precedence, and exclusivity constraints • Resource constraints limit the number of investments • Precluded investment may be due to overlap in functionality or competition for non-cash resources • Convert the multi-criteria resource allocation problems into integer programming maximization-type problems

  14. The IT Investment Model (5/7) • Structuring the AHP hierarchy

  15. The IT Investment Model (6/7) • AHP theory • An overall view of the complex relationships • Help the decision-maker assess the importance of the issue • Support meaningful comparisons between attributes • Steps of using • Establish the decision hierarchy • Create input data and make paired-comparisons of the decision elements • Estimate the relative weights of the decision elements • Aggregate the relative weights of decision elements to arrive at a final set of ratings • For the decision alternatives

  16. The IT Investment Model (7/7) • Incorporating quantitative and qualitative investments • In practice and theory • No consensus on the appropriate mechanism for ranking IT investments • Objective evaluation method • Net present value, cost-benefit analysis, project risk, value analysis, benchmarking, multiple criteria approach, DSS evaluation, aggregate scoring technique, and anecdotal evidence • Subjective method • Attitude surveys and the opinions of users and analysts

  17. An Information Systems Example (1/4) • A simple hierarchy illustration • Includes both financial and non-financial criteria • Compare on the basis of corporate strategies • Investment risk • Revenue enhancing • Operating efficiency • Customer satisfaction • Market growth

  18. An Information Systems Example (2/4) • Steps • Define the decision hierarchy • The goal is to rank the decision alternatives • Input the data • Expert ChoiceTM • The input data are manipulated using matrix algebra to produce the relative weights or priorities • Aggregation of all weights to produce a vector of composite relative weights between the criteria and the alternatives

  19. An Information Systems Example (3/4)

  20. An Information Systems Example (4/4) • Optimizing using integer programming • Maximize the AHP priority weights with the resource constraint • The optimal solution is (1,1,0,1,0) • The objective function value is equal to 0.709 • Higher values signify higher overall returns for the IT investments

  21. Evidence From Two Case Studies (1/7) • Research methodology • Two case studies • Use the IT investment model • Contextual conditions could impact the outcomes • The goals • Ascertain the efficacy of the proposed ranking mechanism • Collect and report the attitudes, behaviors, and perceptions • Results were reviewed by the CIOs with minor corrections and revisions • Use multiple cases • Allow the investigator to replicate the results and improves generalizability • The study will show • Management involvement is necessary • Organizational structure affects the success of the ranking process • Hot and Lukewarm

  22. Evidence From Two Case Studies (2/7) • Case study background of companies • Two U.S. utility companies • North-central region and southern region • Similarities • Generators of electricity, retail and wholesale markets, sold surplus power, and controlled their own transmission and distribution systems • Both had CIOs committed to IT planning

  23. Evidence From Two Case Studies (3/7) • Hot • Smaller company under greater competitive pressure • Highly participative management structure with younger management • Previous experience in non-regulated industries • Highly committed to planning and the strategic use of IT • Lukewarm • Relatively secure markets • Issues of deregulation • Shortly put markets under competitive pressures • With traces of political rivalry • Top management was without experience outside their field • CEO and CIO had previous experience in non-regulated industries • Committed to planning and increasing returns on IT investments

  24. Evidence From Two Case Studies (4/7) • IT planning and evaluation - Hot • Interest in IT planning and using IT strategically • Want a system • Satisfy all areas of management • Ask IT management for assistance in identifying technologies • That might allow revision of business processes to improve efficiencies and customer service • A combination of project evaluation tools • ROI, payback, and a corporate model • Useful but probably unreliable

  25. Evidence From Two Case Studies (5/7) • IT planning and evaluation - Lukewarm • Delegate all IT planning to the CIO • Complain about the time and cost of implementing systems • IT steering committee • Composed of several senior managers • Rely heavily on the opinion of the CIO • The IT plan contained • A wish list of applications that continually changes with the political climate • Use a cost/benefit and payback approach • Selection of projects depends on • How well managers could creatively assign dollars to benefits

  26. Evidence From Two Case Studies (6/7) • The Hot results • The decision criteria and sub-criteria • Originally developed by a team of IT managers • Later modified by other members of management • The participators were familiarity with AHP prior to the session • Use a modified Delphi technique to decide the weights • IT management played an impartial advisory role • The initial analysis was completed • Working with managers from finance, engineering, and marketing • Use both the AHP and integer programming models • Disadvantage • Total time involved in making the paired-comparisons and estimating other parameters • Advantage • Their understanding of the process would help to make future estimates easier and cut the time requirement • Select five IT investments with a capital requirement in excess of $18.5 million

  27. Evidence From Two Case Studies (7/7) • The Lukewarm results • Expected to benefit from the results of the Hot experience • Partly implemented and with less success • Less efficient session • A cross-functional management team • Review and refine the comparisons after individual discussions with managers • The team would have final authority • The CEO supported the process but didn’t participate directly • On the advice of the team • The investments identified as strategic, high cost, and high risk were evaluated • 26 investments were analyzed • Many were overlapping and mutually exclusive • 8 investments were selected with a capital cost in excess of $34 million • Problems • Many managers continually requested revisions of the management team • Use a spreadsheet program • Perform a modified ROI analysis on the selected projects • IT managers felt • The direction was an improvement

  28. Results of Investment Decisions (1/5) • Acceptance • Managers form both companies • Enthusiasm • The documentation for the methodology improved their understanding and made it easier for new managers to grasp • Hot • The internal environment and organizational structure are more conducive to acceptance of new processes • Lukewarm • Acceptance of the methodology had removed a major burden from IT planning • No longer incurred the wrath of managers who had not been funded • This supports • One of the benefits of the MOMC approach is the balancing of conflicting objectives of different users and stakeholders

  29. Results of Investment Decisions (2/5) • Status of the IT investments selected • There was no immediate pressure to cut capital investments • Hot • Lower earnings-per-share • Delay one project to conserve cash and deploy resources to the other projects in order to realize the benefits more quickly • All of the projects were on or under schedule and under budget • Lukewarm • Benefit from reduced political tensions • Most of the projects were on schedule and within budget • The delayed project had suffered from a political tug-of-war about infrastructure issues • IT projects were an outstanding success

  30. Results of Investment Decisions (3/5) • Status of selection process • Hot • Managers were continuing to modify and enhance the model • They wanted to be able to analyze individual investments on a stand-alone basis • The use of a program to quickly generate an initial set of paired-comparisons • Two strategic categories emphasized on valuation of intangible benefits • Tow over $1 million categories emphasized on risk analysis • Lukewarm • The CEO had to contend with several presidents of the operating companies • Less time to focus on IT • Time period was not sufficient • Little had been accomplished towards improving the process, primarily documentation of the process and the training of new managers • The CIO was confident • The next round of investment proposals would be handled more expeditiously

  31. Results of Investment Decisions (4/5) • Generalizable findings • In one firm • The CEO had greater knowledge of IS • The CEO worked closely with the CIO and other managers followed the lead • In the other firm • The CEO had superficial knowledge • The CEO did not work closely with the CIO • Hot had capitalized on the new process to insure success and reduce the time requirement on management • By extending the model and adding administrative controls • Lukewarm accomplished less • Managers in both firms had an improved attitude • The new process improved the quality of information available to measure investment proposals, increased the involvement of managers, and added credibility to the final results • An investment’s potential return may be reduced • Because of implementation problems • The inability to control quality during software system development

  32. Results of Investment Decisions (5/5) • Summary

  33. Discussion (1/2) • Benefits • The ability of the model to handle a large number of criteria • The ability to represent both tangible and intangible items • The ability to model exclusivity and dependency of investments • The ability to quickly reflect revisions • The explanatory power of the model • The support for group decision-making • Limitations • The lack of a financial measure of profitability • The overall time requirements for management • The problem of valuing intangibles, although ameliorated, remained

  34. Discussion (2/2)

  35. Study Contributions (1/2) • Provide a tested process for prioritization and selection of IT investments • Identify benefits and limitations inherent within the process • Identify facilitators and inhibitors and generalizable findings to the approach • Assist the introduction of the process

  36. Study Contributions (2/2) • Suggestions for future research • Further case studies • Suggested from different industries • Provide more insights into the completeness of the approach • Examine the impact of contextual variables on the success of the IT investment model • The balancing of investment risk was not tested in this study • The relationship between process credibility and subsequent development and implementation remains unresolved

  37. Conclusions • The MOMC approach merits attention as a investment selection and ranking tool • Utilize AHP and integer programming • Improve the IT investment process • Strictly quantitative approaches have not yielded satisfactory results • Subjective approaches lack explanatory power and can not be easily adjusted to reflect new knowledge • Basing selection criteria on business strategies ensures the alignment of IT investments with these strategies

More Related