Get Results From Your Performance Audits

1. Get Results From Your Performance Audits

2. Session Objectives This session is intended to help you: • Identify and select high impact performance audits. • Perform preliminary reviews that enhance audit effectiveness and efficiency with: • The audit team having an understanding of the program or entity being audited. • A prioritized assessment of audit risk and audit opportunity relative to the program or entity. • A basis for developing audit objectives or terminating the audit.

3. Session Objectives This session is intended to help you: • Prepare preliminary review documents that present a persuasive rationale for proceeding with or terminating a performance audit. • Develop well-defined audit objectives to achieve a focused audit effort for maximum audit efficiency. • Understand how group activity techniques add value to performance auditing.

4. What Is a Performance Audit? A Performance Audit is an analysis that answers a question. The question is: “What Is?” Which is then compared to: “What Should Be?”

5. Yellow Book Description Performance audits are defined as audits that provide findings or conclusions based on an evaluation of sufficient, appropriate evidence against criteria. Performance audits provide objective analysis to assist management and those charged with governance and oversight in using the information to improve program performance and operations, reduce costs, facilitate decision making by parties with responsibility to oversee or initiate corrective action, and contribute to public accountability. (2.10)

6. Examples of High Impact Performance Audit Findings • Unemployment Insurance Agency • Business Application Modernization (BAM) Project • Pharmaceutical Costs

7. Unemployment Insurance Benefit Overpayments and Nonmonetary Eligibility Determinations We estimated that UIA failed to identify and pursue recovery of overpayments of up to $72.5 million: Overpayments - • Claimant wage and benefit cross match process = $17.9 million • Cross-program benefit offset process = $12.1 million • Verification of claimant identity = $8.2 million • Deceased claimants = $0.4 million • Untimely nonmonetary eligibility determinations = $26.0 million • Ineligible alien claimants = $7.9 million • Total = $72.5 million

8. Unemployment Insurance Benefit Overpayments and Nonmonetary Eligibility Determinations We estimated that UIA did not assess fraud-related penalties of between $120.0 million and $236.6 million: Unassessed Penalties- • Classification of claimants’ misrepresentations = $81.5 - $191.8 million • Claimant wage and benefit cross match process = $37.8 – $43.4 million • Deceased claimants = $0.7 – $1.4 million • Total = $120.0 - $236.6 million

9. Business Application Modernization (BAM) Project Enforcement of Contract Provisions - The Department should have collected liquidated damages resulting from the development contractor failing to meet the agreed-upon implementation dates for BAM. The development contract specified that liquidated damages would be assessed at $200,000 per month if the contractor failed to meet the agreed-upon implementation dates for Phases 3A, 3B, and 3C. The development contractor missed the initial implementation date of August 2007 for Phase 3A (renamed as Release 1). As a result, we calculated that the development contractor owed the State liquidated damages of $12.5 million.

10. Prisoner Pharmaceutical Costs

11. Prisoner Pharmaceutical Costs

12. How to Identify and Select High Impact Performance Audits?Start with a Risk Ranking Qualitative Quantitative Program/System

13. Public Health, Safety, Protection • Service Delivery • State Security • Information Security & Privacy • Economic Growth • Education • Conservation & Protection of Natural Resources • Commercial or Public Transportation Qualitative Risk RankingIs there a risk to:

14. Public Health, Safety, &Protection Injury or loss to life, care & protection of vulnerable citizens, privacy or citizen rights, food safety, agriculture Service Delivery Services not factored into other categories (examples, state park reservation system, hunting permits, licensing), Medicaid payments provided to doctors, eligibility determination, family reunification services, mental health services, etc. State Security Homeland security, emergency preparedness (state & citizen), customs & borders, national guard, escape, recurring offender, parole violations, and MSP. Information Security & Privacy Protection of IT resources, data security Qualitative Risk RankingThings to consider:

15. Economic Growth Michigan economy growth, job increase, business increase Education K - 12, higher-ed, adult ed, early childhood, special ed, laws & regulations, school aid, school improvement grants Conservation and Protection of Natural Resources Air, land, waste, water, forests, wildlife Commercial or Public Transportation Roads, rail, public transportation, bridges, ferries, aeronautics Qualitative Risk RankingThings to consider:

16. Qualitative Risk Ranking 0 = Not Relevant 1 = Low Relevance 2 = Moderate Relevance 3 = High Relevance

17. Quantitative Risk RankingIs there a risk of: • State Revenue Sources Not Being Realized or Collected • Ineffective and/or Inefficient Use of State Resources • Improper Payments (State or Federal) • Contingencies of Potential Liabilities • Value/Condition of Major State Assets

18. Value of Major Assets Need for replacement, need for repairs, and need for maintenance using State resources Examples of Assets: infrastructure, bridges, roads, property, IT equipment State Revenue Sources Not Being Realized or Collected Failure to collect tax revenues or fees, incorrect federal match, underreporting of accounts receivable Contingencies or Potential Liabilities Financial obligation, debt, claim Quantitative Risk RankingThings to consider:

19. Quantitative Risk RankingThings to consider: Ineffective and/or Inefficient Use of State Resources Not seeking federal grants, higher priced contracts, use of overtime, staffing allocations, potential privatization, combined departmental operations, program does not function as intended, program provides no value to State or stakeholders, ineffective program Improper Payments (State and Federal) Payments to ineligible recipient, payments for ineligible service, duplicate payments, payments for services not received, wasted payments that should not have been made, payments of incorrect amount (overpayment & underpayment)

20. Quantitative Risk Ranking 0 = Not Relevant 1 = Less than $5 million 2 = $5 million - $10 million 3 = $10 million - $50 million 4 = Over $50 million

21. Program/System Level Risk RankingIs there a risk of: • Unauthorized Access of Loss of Sensitive Information • Persons/Clients at Risk if Program Fails to Provide Services • Internal Monitoring/Oversight • Reduced Confidence in Government • Legislative/Media Interest • Material Control Weaknesses • Reliability of External Data

22. Reliability of External Data For example - daycare billings from providers, data from school districts, Medicaid provider billings. Lack of oversight of third party administrators. Program Environment Unreliable decision-making data, competent staff, management oversight, tone at the top, high staff turnover, limited resources, segregation of duties, limited evaluation of program data Internal Monitoring/Oversight Functioning internal audit Program/System Level Risk RankingThings to consider:

23. Persons/Clients at risk of program fails to provide services Clients not receiving assistance, medical treatment, or needed services; persons harmed, neglected, or killed because of failure to keep safe or protect Unauthorized access or loss of sensitive information Confidentiality of information, potential impact to organizational operations, assets, or individuals Reduced confidence in government Past program failure Program/System Level Risk RankingThings to consider:

24. Legislative/Media Interest Recent or current legislative interest Material Control Weaknesses Lack of prior audit finding correction, demonstration to resolve material control weaknesses, progress to strengthen controls to address material control weaknesses Other Unusual risks such as fraud, significant safety risks, other extreme high risks not fully measured elsewhere in model Program/System Level Risk RankingThings to consider:

25. Program/System Level Risk Ranking 0 = Not Applicable 1 = Low Risk 2 = Medium Risk 3 = High Risk

26. Audit Risk Ranges High Risk Audit = 77 and Up Moderate Risk Audit = 39-76 Low Risk Audit = 0-38

27. How do I begin a Performance Audit? To determine the SCOPE of your performance audit, start by gathering INFORMATION and obtaining an understanding of the program or entity.

28. INFORMATION How Do You Get It?

29. NLPES SURVEY NASACT

30. Exercise 1 As auditors, we are used to being observant. But just How Observant Are You? • On a standard traffic light, is green on the top or bottom? • When you walk, does your left arm swing with your right or left leg? • On the U.S. flag, is the top stripe red or white? • Which way does water go down the drain: clockwise or counter-clockwise? • Which way do fans rotate? • Whose face is on a dime? • Do books have even numbered pages on the right or left side? • How many curves are there in a standard paper clip? • How many lug nuts are on a standard car wheel? Great Job!!

31. Determine the Scope and Gather Information During the Preliminary Review • The primary goal during the preliminary review is to gain the best understanding of the agency/program and its processes. This will allow the audit team to identify deficiencies and risks, and plan the audit scope and methodology.

32. Preliminary Review Process • Obtain an overall understanding of the audited activity. • Quickly gather information without detailed verification. • Gain an understanding of internal control. • Identify audit risk and audit opportunity.

33. Preliminary Review Process • Audit Risk • The degree to which activities are exposed to the potential for ineffective or inefficient use of resources; the inappropriate disclosure of data; or to forms of fraud, waste, and abuse which could result from inadequate controls.

34. Preliminary Review Process • Audit Opportunity • The potential for developing audit findings related to effectiveness and efficiency of a program.

35. Preliminary Review Process • Preliminary Review Considerations • What are the program’s mandates, responsibilities, functions and activities? • Which functions are most critical to the overall operation of the program? • What are the risks related to the program? • What responsibilities involve/create the most problems? • What changes would make your program more effective in accomplishing its designated responsibilities? • Who are the program’s clients/customers? • Does program management have any concerns that should be considered? • Have recent reorganizations occurred (downsizing)?

36. Preliminary Review Process • Preliminary Review Procedures/Methodologies • Interview program personnel. • Perform analytical reviews of expenditures, revenue, and data pertinent to the program’s mission. • Analyze prior audit results of audits of similar programs in other states. • Review other external or internal audits. • Review program mission, goals, objectives, policies, & procedures. • Obtain organization charts and review staffing levels, including the number and type of employees. • Research professional literature. • Tour facilities, including off-site locations. • Review other states’ similar programs and their mission, goals, objectives, and requirements. • Obtain norms and standards pertinent to the program’s operations.

37. Preliminary Review Summary Preliminary Reviews enhance audit effectiveness and efficiency with: The audit team having an understanding of the program or entity being audited. A prioritized assessment of audit risk and audit opportunity relative to the program being audited. A basis for developing audit objectives. Achievable audit budgets.

38. Developing Audit Objectives What is the audit going to accomplish? Specific audit objectives are critical to the process of auditing smarter and focusing our audit efforts. Without precisely stated objectives, the risk is that the audit work will not produce the desired results. Therefore, it is important to define the issue, problem, or concern that the audit is to examine.

39. Types of Objectives Effectiveness • Success in achieving mission and goals Efficiency • Achieving the most outputs and outcomes practical with the minimum amount of resources.

40. Audit Objective Characteristics Clarity • Objectives should be stated clearly to inform the reader what aspects of the program, activity, or function are being assessed. Neutrality • Objectives should be stated in neutral terms so that the reader understands that we gathered and analyzed data without bias.

41. Writing Solid Audit Objectives • Objectives should be: • Specific • Focused on one topic • Objectives should not be: • Broad • Compound

42. Writing Solid Audit Objectives Ask Yourself: • Will I be able to conclude on this objective? • Will I have multiple conclusions ? • Effective at one thing; not effective at another thing. • Will my conclusion make sense? • Is this something we even want to conclude on? • We may not want to conclude that tax collections were accurate, but we would conclude that their efforts to collect taxes were effective.

43. Audit Objectives - Example 1 To assess the effectiveness of the Department's efforts to establish performance measures, identify best practices, and evaluate services provided to adults transitioning from foster care to self-sufficiency.

44. Audit Objectives - Example 1Final Report Version • To assess the effectiveness of DHS's efforts to evaluate services provided to youth transitioning from foster care to self-sufficiency. • To assess the effectiveness of DHS's efforts to ensure the propriety and reasonableness of discretionary payments and post-secondary educational assistance payments made on behalf of youth transitioning from foster care to self-sufficiency. • To assess the effectiveness of DHS's efforts to ensure that training for caseworkers is tailored to address youth transitioning from foster care to self-sufficiency.

45. Audit Objectives - Example 2 To assess the effectiveness of the Correctional Facility's efforts to comply with policies and procedures.

46. Audit Objectives - Example 2 Final Report Version To assess the effectiveness of DOC's efforts to comply with selected policies and procedures related to safety and security at the St. Louis Correctional Facility.

47. Audit Objectives – Example 3 To assess the effectiveness of the Bridges information system.

48. Audit Objectives – Example 3Final Report Version • To assess the effectiveness of DHS and DTMB's efforts to implement change controls over the Bridges application and data. • To assess the effectiveness of DHS and DTMB's efforts to implement controls to ensure the accuracy, completeness, and timeliness of Bridges interfaces.

49. Audit Objective Summary • Well-defined audit objectives achieve a focused audit effort and maximum audit efficiency.

50. Referent Group Meetings Objectives Documents to Bring Topics Discussed