1 / 30

Auditing Large and Complex IT Projects

Auditing Large and Complex IT Projects. Auditor General State Of Florida. Overview. Basic IT project risks. Example IT projects in Florida. Issues and challenges to auditing large IT projects. Checklist of suggested audit questions. Suggestions for further reading. Basic IT Project Risks.

demont
Download Presentation

Auditing Large and Complex IT Projects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Auditing Large and Complex IT Projects Auditor General State Of Florida

  2. Overview • Basic IT project risks. • Example IT projects in Florida. • Issues and challenges to auditing large IT projects. • Checklist of suggested audit questions. • Suggestions for further reading.

  3. Basic IT Project Risks • Planning. • Procurement. • Contract management. • Development and Implementation. • Post-implementation.

  4. IT Project Risks - Planning • Lack of clearly defined goals, objectives, and requirements. • Lack of identification of project risks. • Poorly defined project management structure. • Unrealistic budget – time and dollars. • Lack of stakeholder & user buy-in. • Extent and impact of business process reengineering not sufficiently addressed. • Inadequate basis for development of procurement criteria. • No established baseline for measuring cost savings

  5. IT Project Risks - Procurement • Poorly defined statement of work. • Lack of basis for competitive solicitation. • Flawed evaluation of proposals. • Litigation after award.

  6. IT Project Risks – Contract Management • Contract terms that don’t adequately protect the state’s interest. • Poorly defined deliverables. • Lack of agreed upon performance measures. • Payment without suitable performance. • Lack of recourse for non-performance or poor performance by contractors. • Vague ownership provisions – data, software, hardware. • Vague or missing provisions regarding data security. • Poor or lacking termination & transition clauses to protect state’s interest should either party cancel.

  7. IT Project Risks - Development and Implementation • Implementation of software that does not function properly & include good controls. • Poor data security. • Inadequate training and knowledge transfer, resulting in lack of sufficient knowledge to operate, maintain, and use the new system • Cost overruns. • Failure to meet implementation deadlines • Over-customization of vendor software making future upgrades costly to obtain and implement. • Loss of data integrity during conversion/cutover.

  8. IT Project Risks – Post Implementation • Security vulnerabilities as a result of security not being hardened after implementation. • Integrity of system compromised through poor maintenance and change control. • On-going reliance on contractors. • In-house knowledge not sufficient to maintain system or infrastructure. • Unable to turn off old systems that were to be replaced. • Unable to sustain user buy-in.

  9. Example Florida Projects • My Florida Marketplace. • People First. • MyFlorida Alliance. • Aspire. • Various educational entities.

  10. MyFlorida Marketplace • A Web-based e-procurement system for state agencies. • Developed by customizing Ariba software. • Application Service Provider – Accenture. • Responsible State agency: Department of Management Services. • Total contract cost : $93.9 million. • Contract term: October 2002 through November 2010. • Funded through 1% transaction fee on purchases.

  11. MyFlorida Marketplace – Planning and Procurement Issues • Cost-benefit and risk analysis not conducted prior to decision to outsource and issuance of ITN. • Insufficient involvement of key end-users and stakeholders in development of ITN. • Lack of significant baseline data for planning and analysis. • No mechanism to capture and track statewide costs associated with MFMP.

  12. MyFlorida Marketplace – Contract and Project Management Issues • System not formally accepted prior to implementation. • Heavy reliance on ASP without sufficient monitoring of performance. • Limited monitoring by DMS. • Third-party monitor’s duties strayed from monitoring ASP.

  13. MyFlorida Marketplace – IT Functionality and Control Issues • Deficient change control process. • System performance and capacity management needed improvement; performance issues existed. • Disaster recovery plan not timely approved and lacked important provisions. • Insufficient back-up provisions. • Deficiencies in security controls.

  14. MyFlorida Marketplace – IT Functionality and Control Issues • Data integrity issues. • Problems with accounting system interface. • Problems with attached scanned documents. • Declining use of system by State agencies. • System functionality issues. • System performance issues. • Workflow inefficiencies.

  15. People First • An HR outsourcing initiative. • Includes a Web-based enterprise-wide ERP system supporting: • HR administration. • Benefits administration. • Payroll administration. • Staffing administration.

  16. People First • Service provider – Convergys. • 9-year contract. • $349.9 million. • People First application built using SAP software. • System phased in between May 2003 and January 2005. • Convergys provides Florida with a SAS 70 report on its service center.

  17. People First – Planning and Procurement Issues • Cost-benefit and risk analyses not performed prior to release of ITN. • Inaccuracies in cost estimates within Business Plan. • No system to track statewide project cost. • Deficiencies in evaluation and negotiation processes.

  18. People First – Contractual Issues • Legal records retention requirements not included in contract. • No provision for State to approve new or changes in subcontractors. • No provision for subcontractors to obtain background checks. • Many deliverables not timely provided. • Additional amounts paid to third-party monitor for performing services already required.

  19. People First – Operational Problems • System functionality problems and errors. • Lack of written security guidelines. • Off-shoring of State employee personnel data. • Planned system components not implemented, requiring workarounds by the agencies.

  20. MyFlorida Alliance • An effort of the State Technology Office to reengineer its IT functions and governance structure through outsourcing many of its primary functions. • STO was responsible for centralized management of IT for the executive branch of Florida government.

  21. MyFlorida Alliance – Functions to be Outsourced • Enterprise e-communications. • Enterprise technology services desk. • Enterprise applications management. • Enterprise data center operations and consolidation. • Enterprise portal, enterprise security, various others planned.

  22. MyFlorida Alliance • Two prime contractors – Bearing Point and Accenture. • Contracts signed August 13, 2003. • 7 year term. • $324.7 million. • Procurement method: ITN. • All agreements since terminated by STO. • STO abolished in law effective 7/1/07.

  23. MyFlorida Alliance • Inadequate planning & documentation to support decisions to outsource or use ITN method. • Deficiencies in proposal evaluation and negotiation, limiting fairness and competition. • Contracts lacked provisions to protect the State and did not pinpoint total cost to State. • Cost savings analyses questionable or not available.

  24. Aspire • Replacement for the State’s general ledger accounting and cash management systems. • Project beginning date - 9/8/2003. • Original planned rollout in three waves of agencies going live, from July 2005 through December 2005. • Rollout schedule amended five times. • Project suspended 5/17/2007, with $89 million spent to date.

  25. IT Projects at Florida’s Educational Entities • University ERP implementations – for example, University of Florida. • Insufficient system testing • Insufficient staff training • Functional system problems • System governance issues • IT security and control issues • Various community colleges and school districts.

  26. Issues and Challenges to Auditing Large IT Projects • When to audit. • During the project? • Postaudit? • Impact of outsourcing. • Authority to audit contractor & subcontractors • Contractor responsiveness to audit requests • Applicability of SAS 70 audits, if available, to non-financial audit objectives • Lack of sufficient entity knowledge of system. • Difficulty in extracting data needed for audit • Availability of entity staff to timely respond to audit requests during implementation or post-implementation.

  27. Suggested Audit Checklist • Planning. • Procurement. • Contract Provisions and Management. • Project Management.

  28. Suggested Further Reading • MyFlorida Marketplace • 2007-076 (IT Audit) • 2006-015 (IT Audit) • 2005-116 (Operational Audit) • People First • 2007-087 (Operational Audit) • 2005-047 (Operational Audit)

  29. Suggested Further Reading • MyFlorida Alliance • 2005-008 • University of Florida • 2006-040 (Operational Audit) • 2006-145 (IT Audit) • Chapter 2007-115, Laws of Florida • Abolishes STO and creates AEIT

  30. Questions?

More Related