Continuous audit: today and tomorrow

1. Continuous audit:today and tomorrow Miklos A. Vasarhelyi KPMG Professor – Rutgers University Senior Consultant- AT&T Laboratories

2. Outline • An evolving framework • Some Key issues / the state of the art • Some CARLAB experiences • Six Steps in Implementing CA • Organizational Context • Opportunities and Challenges • Conclusions

3. An evolving audit framework

4. Report level Assurance Data level Assurance Process level Assurance Assurance of Key Processes Assurance of Data elements Assurance of Reports An evolving audit framework • XML/ XBRL datum • Generated and modified by different processes • Balkanization of data • Control / Assurance tags • Compliance reports becoming commonplace • Traditional audit is an instance of RLA • Generated and modified by different processes • Process reviews a la Systrust • Internal or outsourced • Third party processes are to become the norm • Intra and Inter process controls an issue

5. An evolving continuous auditframework Continuous Audit Continuous Audit Continuous Control Monitoring • Automation • Sensoring • ERP • E-Commerce Data CA = CCM+ C(D)A CA -> Continuous Audit CCM -> Continuous Control Monitoring C(D)A -> Continuous Data Assurance

6. Some Key Issues • Two recent surveys (ACL and PWC) show that a large number of key companies are attempting to perform continuous audit like functions • An industry of software is evolving with ACL, IDEA, APPROVA, and others growing rapidly • Control Monitoring and Continuous Data Assurance are the main approaches • The first recorded application was AT&T Bell Laboratories CPAS effort in the 1986-1991 period • The Rutgers CarLab is working in leading applications

7. Continuous Auditing Value Proposition • Improved business performance • Innovations in information technology & analytical modelling enable: • More frequent, timely, accurate & relevant business performance information • Lower compliance risk • Cost reduction

8. CAR-Lab Experiences • Control monitoring at Siemens • Transaction monitoring at Unibanco • Continuous (data) assurance at HCA • Other • Conceptual developments • Simulating Liberty • EBR work • KPMG projects

9. Overview of CaR-Lab examples

10. Expanded Audit Coverage Significant Cost Savings Siemens' – Project Value Proposition Automated Business Process Controls Monitoring Project

11. Siemens' – Project Features • Formalize & automate internal audit procedures used for business process controls monitoring • Conduct “man vs. model” assessments • Calibrate “exception rules” to optimize model performance • Scale up to all SAP instances • Increase frequency of model application, where feasible • Transition to Approva application and extend the model where optimal

12. A 3 pronged approach to audit automation • Automate audit plan using delivered Rule Sets: Est 25% of a typical manual audit plan • Automate using external data sets (Static & Variable): Est an additional 25% a typical manual audit plan • Re-enginer manual controls into automated controls with improved control precision: Est an additional 25% a typical manual audit plan • Total = Automation Opportunity ~75%!!

13. Auditor Management Audit Parameterization Tool Other Static Parameters Deter- ministic Stocha- stic External Table comparisons Snapshot comparisons Other Data Extraction Remote Audit Communic. Tool Interactive Mail Management Tool Sustainable Object Verification Tool Other MCP Audit Evidence Receptacle Master Audit Program Operating Alarm Flows Operating Alarm Flows CA Control Dashboard A.A.S (audit Action Items) From Siemens Approva and other literature Inference Engine Evergreen Opinion Class of Auditable Actions ---- of Audit Processes

14. IT / IA Continuous Auditing Program at Unibanco

15. Unibanco – Some CA Program Features • Automated monitoring of over 5 million customer accounts on a daily basis using 25 automated procedures to: • Detect errors • Deter inappropriate events & behaviors • Reduce or avoid financial losses • Help assure compliance with existing laws, policies, norms and procedures • Examples of “low hanging fruit:” • Customer advances • Excess over credit limit • Returned checks • Federal tax payment cancellations • TED emissions (should this be omissions?)

16. Unibanco – Advances to Clients Monitoring

17. Continuous Data Assurance (CDA) at a Major Health Services Provides (HSP) • HSP is a large national provider of healthcare services, composed of locally managed facilities that include numerous hospitals and outpatient surgery centers. • IT internal audit provided access to unfiltered extracts from their transactional databases, comprising all procurement cycle daily transactions from October 1st, 2003 through June 30th, 2004: Over 500,000 data points. • Dataset mimics what a CDA system has to deal with: highly disaggregate data flowing through CA system in real time. • Audit procedures have to be developed for this environment.

18. Analytical Procedures in CA • Analytical procedures used in the planning, substantive testing, and reviewing stages of an audit. We focus on substantive testing. • In conventional auditing first apply analytical procedures to identify potential problems, Then, focus detailedtransaction testing on the identified problem areas. • In CDA the sequence isreversed: • Use automated generaltransactiontests to all the transactions and filter out identified exceptions for resolution. • Apply automated analytical procedures to the filtered transaction stream to identify unforeseen problems. • Alarm humans to investigate anomalies.

19. Continuous Data Assurance • Automation of Transaction Testing: • Formalization of business process rules as transaction integrity and validity constraints. • Verification of transaction integrity and validity  detection of exceptions  generation of alarms. • Automation of Analytical Procedures: • Selection of critical business process metrics and development of stable business flow (continuity) equations. • Monitoring of continuity equation residuals  detection of anomalies  generation of alarms.

20. Continuous Data Assurance System Automatic Analytical Monitoring: Continuity Equations Automatic Transaction Verification Anomaly Alarms Exception Alarms Responsible Enterprise Personnel Business Data Warehouse Enterprise System Landscape Materials Management Sales Ordering Accounts Receivable Human Resources Accounts Payable

21. Establishing Data Integrity: A Procurement Example • Referential integrity along the business cycle and identification of completed cycles: P.O.  Shipment receipt  voucher payment. • Identification of data consistency issues and automatic alarms to resolve exceptions: • Changes in purchase order vendor numbers; • Discrepancies between the totals and the sums of line items; • Discrepancies between matched voucher amounts.

22. Detection of Exceptions • Referential integrity violations • PO without matching requisition • Received item without matching PO • Payments without matching received items • Data integrity violations • PO has zero order quantity • Received item has negative quantity • Invalid payment check numbers (e.g. All 0s) • Gross payment amount is smaller than net payment amount

23. Continuity Equation Based CDA • Continuity Equations: • Stable probabilistic models of highly disaggregated business processes, uses as the expectation models for process based analytical procedures. • Originated in physical sciences (various conservation laws: e.g. mass, momentum, charge). • Continuity equations are developed using statistical methodologies of: • Linear regression modeling (LRM); • Simultaneous equation modeling (SEM); • Multivariate time series modeling (MTSM) using various Vector Autoregressive Models (VAR).

24. Basic Procurement Cycle t2-t1 P.O.(t1) Receive(t2) t3-t2 Voucher(t3)

25. Ideal Continuity Equations of Basic Procurement Cycle Receive(t2)= P.O.(t1) Voucher(t3)= Receive(t2) • Aren’t partial deliveries allowed? • Are all orders delivered after exactly the same time lag? • Are there any feedback loops?

26. Estimated Continuity Equations of Procurement Using VAR Model P.O.(t)= 0.24*P.O.(t-4) + 0.25*P.O.(t-14)+ 0.56*Receive(t-15) + εPO Receive(t)= 0.26*P.O.(t-4) + 0.21*P.O.(t-6)+ 0.60*Voucher(t-10) + εR Voucher(t)=0.54*Receive(t-1) - 0.17*P.O.(t-9) + 0.22*P.O.(t-17) + 0.24*Receive(t-17)+ εV

27. Detection of Anomalies • Anomalies are detected if: • Observed P.O.(t) < Predicted P.O.(t) - Var or • Observed P.O.(t) > Predicted P.O.(t) + Var • Similarly for: • Receive(t) • Voucher(t) • Var = acceptable threshold of variance. • If there is anomaly  generate alarm!

28. Measuring Anomaly Detection • False positive error (false alarm, Type I error): A non-anomaly mistakenly detected by the model as an anomaly. Decreases efficiency. • False negative error (Type II error): An anomaly failed to be detected by the model. Decreases effectiveness. • Detection rate is used for clear presentation purpose: The rate of successful detection of seeded errors. • A good analytical model is expected to have good anomaly detection capability: low false negative error rate (i.e. high detection rate) and low false positive error rate.

29. Simulated Error Correction • Access to highly disaggregate data in real time makes it possible for CA system to detect, investigate and correct anomalies also in (nearly) real-time. • Real-time error correction enables utilizing the corrected rather than the erroneous data in revised continuity equation benchmarks. • Real-time error correction is likely to benefit future anomaly detection. We investigate the magnitude of this benefit using simulation. • Error correction raises important issues about auditor independence, and the line between auditing and monitoring of business processes.

30. Benefit of Real-time Error Correction: MTSM

31. Takeaways from HSP Study • Various statistical methods can be used to derive expectation models of acceptable quality. • But key is access to highly disaggregate data, not which benchmark is used. With such data, most reasonable continuity equation models give usable results. • Real-time error correction significantly improves error detection. • More disaggregated models are not always better: weekly data can be more stable than the daily one. • Alarms have to be managed – trade-off between Type I and Type II errors.

32. Implementation Issues in CA

33. Background • While technologies of continuous audit have been extensively discussed and are progressively emerging the more mundane issues of their implementation in a socio-technical environment have been neglected • http://www.theiia.org/itaudit/features/in-depth-features-2-10-08/feature-2/

34. Priority • Areas 2. Rule 6. Action and Reaction 3. Frequency Audit Control Panel 5. Follow-up 4. Parameterization Six steps of process implementation

35. 1. Identification of Priority Areas • Modularize risk areas, rate these risks and evaluate the cost x benefits • Identify the basic audit objects • Choose critical business processes that will be the focus of continuous audit (low hanging fruit) • Identify key data in for the implementation of Continuous Audit in the mapped processes • Political Considerations

36. Key Objective of Audit Procedure • Detective • Deterrent • Financial • Compliance

37. 2. Rules of Monitoring and Auditing • Once an area of CA is chosen the “rules” of monitoring, alarming, and assurance must be established • These must take into consideration the legal and environmental issues as well as the objectives of the particular process • The CA process is established adopting certain rules, frequencies, and parameters. • e.g. we will monitor bank accounts in overdrafts or in excess limits

38. 3. Frequency • The natural rhythm of the process • Timing of computer processes • Timing of business processes • Cost benefit considerations • Nature of procedure objectives • Deterrence • Prevention

39. 4. Parameterization • Define parameter to analyze in accordance with the risk • eg.: Monitoring all accounts in overdrafts in daily basis , that have a balance of debt 20% larger than its limit of loan and bigger than 1000 USD

40. 5. Follow-up • Who will receive the alarm? • Management? • Audit leadership? • Immediate superior of the responsible for the data • The timing of the follow up • Pass the alarm along immediately • Reconcile the alarm prior to follow up • Wait for 3 sequential days of similar alarms to follow up • Escalation guidelines • E.g. after three days send to the immediate superior’s superior or wait for 3 days prior to the re-escalation

41. 6. Action and Reaction • Guidelines for dealing with auditees • Lack of bias • Consistency of response • Guidelines for individual factor considerations • Concern with collusion

42. Organizational Issues

43. Organizational Structure for CA • Is CA a part of the audit function or ofmanagement? • Its part of the audit function • Should there be a separate continuous audit group? • Yes, to facilitate its implementation progressively in the many areas of continuous audit

44. Workforce Effects • Progressively labor requirements for the traditional audits supported by CA will reduce and deeper audit will become possible • Rebalancing of workforces • High technological competencies needed

45. Opportunities and Challenges

46. Opportunities for business and research (1) • Control system measurement • We are in a pre-paradigmatic stage of control documentation and measurement • We do not know how to monitor controls in large ERPs • We do not know how to provide a really supportable opinion on controls • We do not know how to rate combinations of controls

47. Opportunities (2) • Business Process Monitoring and Alarming • Auditors have to carve a position on the new monitoring and control environment • Auditors can collect exception “alarms” as trusted parties and incorporate these into evidentiary matter • Auditors can be “trusted”

48. Opportunities (3) • Automatic Confirmation Tools • Confirmations will have an increased evidentiary role with eventual elimination of population and integrity worries • Intelligent confirmatory tags can do much • Database to database hand-shaking will be medium • Business opportunity for auditors

49. Opportunities (4) • Audit bots (agents) • Many of the basic audit functions can be emulated by software • These must be eventually developed by the profession to work hand-in-hand with human auditors in the new audit world • These agents will work on all areas including: 1) audit planning, 2) analytical reviews, 4) confirmations, and )5 evergreen opinions

50. Opportunities (5) • Collecting forensic trails • Auditor “black” box • Publishing real-time authenticated reports for different compliance masters • Publishing FD independent compliance reports