Audit principles
This presentation is the property of its rightful owner.
Sponsored Links
1 / 232

Audit Principles PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Audit Principles. NERC Auditor Training Introduction to Audit Principles and Techniques. REMG Compliance Auditor Training. Please Remember…. This is not technical training. There are “hard” and “soft” skills to auditing

Download Presentation

Audit Principles

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Audit principles

Audit Principles

NERC Auditor TrainingIntroduction to Audit Principles and Techniques

Remg compliance auditor training

REMG Compliance Auditor Training

Please remember

Please Remember…

  • This is not technical training. There are “hard” and “soft” skills to auditing

  • This course is focused on auditing skill building and is based on years of auditing knowledge.

  • ALL auditing follow the same basic principles. We ALL need to think like auditors.

  • Discussion is welcomed – especially on how to apply standard practices to NERC compliance audits

  • Training materials are under development, so your feedback is very much appreciated!

  • Trainers assume that you have read the CMEP



  • 10.00 Introductions/Course OverviewAll

  • 10.30 Module 1 - Conceptual FrameworkRuss Hissom

  • Dan Skaar

  • 11.15 Module 2 – Audit Process and

  • PlanningRuss Hissom

  • 12.00 Lunch

  • 1:00Module 3 – Managing RelationsCarol Arneson

  • 1:30 Module 4 – Audit Documentation and

  • Evidence (and RSAW Breakout)Russ Hissom

  • Kevin Goolsby

  • 2:30Module 5 – Workpaper Preparation and

  • Information RequestsCarol Arneson

  • 3:00Break

  • 3.45 Module 6 – Audit Testing and TestingMethodologiesRuss Hissom

  • 4.45 Questions and Discussion

  • 5.00Happy Hour!!



  • 8.00 Module 7 – Interview TechniquesCarol Arneson

  • 9:00 Module 8 – Leveraging Project Management

  • Tools and Techniques for Audit SuccessCarol Arneson

  • 9:30Break

  • 9:50 Module 9 – Report and Workpaper Review Russ Hissom

  • 10:20 Module 10 – Conflict Resolution and Escalation ProtocolsCarol Arneson

  • 10.45 Module 11– Audit Close Activities, Lessons

  • Learned and On-going Performance

  • Management Russ Hissom

  • CloseQuestions and Discussion – Course Evaluations

Compliance program assistance leadership team

Compliance Program Assistance Leadership Team

Virchow Krause Team Members

Carol Arneson, PMP, MBA, Senior Manager in the Energy and Utilities Group, has worked in the utility industry since 1976 specializing in financial and operational needs. She has broad utility experience at two Fortune 500 utility companies where she managed financial, cost management, strategy and business planning, generation support processes, and various other business processes for over 20 years. Carol has managed numerous projects serving municipal and investor-owned utilities including contract compliance audits, energy management services contracting and performance audits.

Russell Hissom, CPA, Partnerin the Energy and Utilities Group, specializes in serving the financial and operational needs of the utility industry. He has extensive experience with financial audits of utilities, management audits for utilities and State Public Utility Commissions, developing utility cost of service and rate design studies, analyzing the input and performance of parties under jointly owned electric generation contracts, assisting with accounting issues under FAS 133/149 and performing operational reviews.

Goals for this course

Goals for this Course




Very brief history of the compliance landscape

Very Brief History of the Compliance Landscape

How Did We Get Here?

Don t underestimate the power that lack of compliance of rules and regulations has on any industry

Don’t underestimate the power that lack of compliance of rules and regulations has on any industry

  • History has shown us that compliance (or lack of compliance) has the power to do many things:

    • Financial collapse large listed companies

    • Reduce market capitalization by billions of dollars for alleged, egregious violations

    • Bankrupt companies

    • Closure of plants

Do you remember

Do You Remember……..

  • Enron (SEC compliance)

  • Worldcom (SEC compliance)

  • Ameranth (SEC compliance)

  • Bear Sterns, Lehman, AIG, Wachovia, Merrill Lynch, etc. etc. (SEC compliance)

  • Southwest Airlines (FAA compliance)

  • Northeast blackout of 2003 (NERC voluntary compliance)

  • Southern Florida blackout of 2008 (NERC mandatory compliance)

Crisis leads to regulation

Crisis Leads to Regulation

  • Blackouts

    • Energy Policy Act of 2005 (section 215 of the Federal Power Act) mandated standards in USA with financial penalties (separate agreements within Canadian jurisdictions)

  • Financial Collapse and Fraud

    • COSO (internal controls)

    • Sarbanes-Oxley (internal controls, governance, civil penalties on corporate officers)

Role of the auditor

Role of the Auditor

  • Auditor’s responsibilities are much more important today than the past given the impacts from non compliance

  • Can you imagine if you, as an auditor, missed a major finding and had you made the finding you could have prevented an incident on the bulk power system? What about Enron, what if the auditor, early in the scandal discovered and reported the irregularities, perhaps things would be very different!

  • ALL auditors’ competency and training must be at a very high level to assure their duties to the industry and maintain the public trust.

  • Auditors’ work must stand up to public scrutiny and legal challenges!

Audit principles

  • What will be next??

Audit principles

  • Questions?

Module 1

Module 1

Conceptual Framework for Auditors

Learning objectives

Learning Objectives

  • Understand what an “audit” is and is not

  • Develop the confidence to perform a competent audit

  • Understand the basic steps in an audit

  • Who are the audit standard setting bodies

  • What kinds of audits are there?

  • What does the technical guidance tell us to do?

What is an audit

What is an Audit?

  • An audit is an evaluation of a person, organization, system, process, project or product

  • It is not an investigation

Audits may not presume a potential violation; investigation presume a potential violation exists. Skills are similar in the conduct of an investigation.

Audits are performed to ascertain the validity and reliability of information, and may include an assessment of a system's internal compliance environment. The goal of an audit is to express an opinion whether some one or some entity meets a “standard” or does not meet a “standard” based upon a systematic review and testing of records. Due to practical constraints, an audit seeks to provide only reasonable assurance that the registrant is compliant with the applicable Reliability Standards.

Necessary skills

Necessary Skills

  • Attention to detail

  • Good understanding of audit risks

  • Ability to work with people and experts

  • Subject matter expertise

  • Deep knowledge of reliability standards applicable to entities being audited

  • Knowledge of government auditing standards that apply to performance audits

  • Task management skills

  • Clear and concise communications

  • Ability to follow a standardized program

  • Good planning skills

  • Team player

  • Willingness to identify issues and be proactive in bringing them to attention

Major steps involved

Major Steps Involved

  • A typical performance audit project involves the following steps:

    • Establish and communicate the scope and objectives for the audit.

    • Develop an understanding of the organization under review. This includes objectives of the audit, measurements, and key requirements. Review pertinent documents and interviews.

    • Identify control procedures used to ensure each key activity type is properly controlled, monitored and documented. Upfront, an internal compliance survey should be completed by the Registered Entity.

    • Develop and execute a risk-based sampling and testing approach to determine whether the most important activities are operating as intended.

    • Report findings and areas in compliance.

    • Complete audit closing tasks, review staff and start to prepare for your next audit.

Tips for success

Tips for Success

  • Be the “ultimate” professional

  • Expect to be “monitored”– lead by example

  • Use empathy – remember what it’s like to sit on the other side of the table. Be compassionate, but firm

  • Remember you’re there to complete a job – not solve the auditee’s problems

  • Be proactive

  • Know the project work planbeforeyou begin

  • Communicate with your audit leader

  • Thoroughly document all testing and findings with quality evidence

Audit types

Audit Types

  • Financial Audit

    • A financial audit is an independent assessment of the fairness by which a company's financial statements are presented by its management

      • Authoritative standard bodies include GAO, Canadian versions as well

  • Compliance Audit

    • A compliance audit is an independent assessment of the compliance by an entity with various laws or regulatory requirements

      • Authoritative bodies include GAO (e.g. chapter 7)

      • NERC compliance audits

  • Management Audit

    • A management audit is an independent assessment of the efficiency in various operating areas by an entity

      • SAS 70 audit, “agreed upon procedures”

Value of audits

Value of Audits

  • Audits are not just checking if things happened or if compliance requirements were met – they can be used to provide great value to the registrant

    • Informal recommendations for process improvements or how to meet compliance requirements are a natural by-product of an audit (orally during the exit interview)

    • Audits serve the public and industry interests; there is a reliance on auditor’s work to identify compliance and non compliance; it’s the responsibility of the entity to comply and take necessary action to be compliance with standards

Auditing concepts techniques

Auditing Concepts & Techniques

  • What is GAGAS?

  • In the United States – this is the standard for government performance audits – Generally Accepted Government Auditing Standards – GAGAS – aka the “yellow book”

  • Standards maintained by the Government Accountability Office (GAO)

  • GAGAS standards incorporate other standard bodies work

  • Requires auditors to serve the public interest and honor the public trust

  • Auditors must perform all duties with integrity, be independent and honest and candid with the entity being audited

  • Auditors should always exercise professional judgment and skepticism

Gao chapter 2 ethical principles

GAO Chapter 2 Ethical Principles

  • Ethical principles guide the work of auditors

    • The public interest

    • Integrity

    • Objectivity

    • Proper use of information

    • Professional behavior

  • Please take five minutes and read Chapter 2.

    Any thoughts?

  • Gao chapter 3 general standards

    GAO Chapter 3-General Standards

    • Independence

      • Free of conflicts

      • Appearance and in fact

    • Professional Judgment

      • Knowledge, skills, experiences,

      • Reasonable care

      • Professional Skepticism

      • Due diligence

    • Competency

      • Blend of education and experience

      • MUST have skills to perform audit

    Auditing concepts techniques1

    Auditing Concepts & Techniques

    • Review GAO Chapter 3

    • What are the key applicable parts of Chapter 3 in the conduct of our audits?

    Professional skepticism

    Professional Skepticism

    • The ability to approach any situation with a skeptical view towards conclusions reached without examining all factual data and using that data to verify and support your conclusion as an audit

    • me!

    Due diligence

    Due Diligence

    Performance Improvement Intent

    Production & Reserve Increases

    Per Person

    Market Share

    Retail Return



    Increase %

    Production Increase

    per Person






    Net Income $mm

    Reserves/10 per


    % ROI


    per Person











    Net Income

    Market Share Change

    Strategic Targets/Initiatives

    % of Capital



    Quality Rating










    Change in

    Market Penetration

    Capital Expenditures

    Natural Gas




    Increase in Gross

    National Product

    Refining Gross Operating






















    The act of researching all available data to support a conclusion or position about an activity or outcome

    Due professional care

    Due Professional Care

    • “Due Professional Care in the Performance of Work”

      • What is Reasonable Assurance?

      • Auditor must plan and perform audit to obtain appropriate evidence so that audit risk is limited to a low level appropriate for expressing an opinion on the assertion tested (making a compliance determination)

      • Absolute assurance may be not possible because of the nature of audit evidence. Point of distinction between an audit and investigation: an investigation may require absolute assurance for prosecution of a violation; therefore, “stacking of evidence” on a potential violation may be appropriate during an audit after a potential violation is discovered .

      • Materiality: is it material? Not all things are the same!

    • Management of the registrant is responsible for assuring compliance to Reliability Standards

    Auditing concepts techniques2

    Auditing Concepts & Techniques

    • GAGAS Continuing Education Requirements (per Government Auditing Standards)

      • Applies to external and internal auditors who perform GAGAS audits

      • Standards require 80 hours every 2 years of continuing education – 24 hours in subjects directly related to the governmental environment or governmental auditing

        • Remaining hours should be in topics that directly enhance the auditor’s professional proficiency to perform audits

        • At least 20 of the 24 hour requirement should be done in a single calendar year

      • Auditors who do not supervise audits or who charge less than 20% of the annual time to audits need 24 hours every 2 years

      • Assume these requirements apply to NERC—stay sharp, be a life long learner in this profession!

    Auditing concepts techniques3

    Auditing Concepts & Techniques

    • Review GAO Chapter 7-Field Work Standards for Performance Audits

    • What are the key applicable parts of Chapter 7 in the conduct of our audits?

      • Audit Evidence

      • Audit Risk

      • Audit Planning

      • Internal compliance environment

      • Sufficiency of audit evidence

    Why do auditor s fail

    Why do Auditor’s Fail?

    • SEC and others have reviewed audits and auditors and have determined several reasons why auditors fail:

      • Failure to obtain sufficient evidence to support conclusions.

      • Failure to maintain independence

      • Failure to follow-up on unusual events (exercising professional skepticism)



    Determine if the audit team is independent

    • “In all matters relating to the audit work, the audit organization and the individual auditor, whether government or public, should be free both in fact and appearance from personal, external, and organizational impairments to independence”

    • If it feels like you’re not independent, you’re not

    • If someone asks you “are you sure you’re independent?” - You’re probably not!

    Other independence matters

    Other Independence Matters

    • Free of conflicts

    • NERC conflict of interest policy

    • Impairment of independence

      • Receiving gifts

      • Favors

      • If you think it is an impairment, it probably is an impairment

    • Must be independent “in appearance” and “in fact”

    Auditing concepts techniques audit evidence

    Auditing Concepts & TechniquesAudit Evidence

    Auditing concepts techniques audit evidence1

    Auditing Concepts & TechniquesAudit Evidence

    • Audit Procedures for Obtaining Audit Evidence

      • Inspection of records or documents

      • Inspection of tangible assets

      • Inquiry

      • Confirmation

      • Recalculation

      • Re-performance

      • Analytical procedures

    Auditing concepts techniques4

    Auditing Concepts & Techniques

    Audit Risk and Materiality in Conducting an Audit

    Inherent Risk (IR) – the risk linked to the activity itself assuming there are no related controls

    • Example: Registrant performs activities linked to the bulk power system operations and planning. Registrant has no documentation and not trained staff to perform requirements under a Reliability Standard

    • Others?

    • ¹A material misstatement under Reliability Standards would be a requirement under a standard is not being met and it’s a material impact to the bulk power system or has the potential to materially impact the reliability of the bulk power system.

    Auditing concepts techniques5

    Auditing Concepts & Techniques

    Audit Risk and Materiality in Conducting an Audit

    Control Risk (CR) - the risk that controls will not prevent, detect and correct errors

    • Example: Registrant has documentation and trained staff, but no evidence of adequate supervision or review.

    • Others?

    Auditing concepts techniques6

    Auditing Concepts & Techniques

    Audit Risk and Materiality in Conducting an Audit

    Detection Risk (DR) – risk that auditor will not detect a material misstatement – function of audit procedure and its application by the auditor

    • Example: Regional Entity sampling of a requirement did not include enough samples. Result was a material number of samples which did not meet requirements (e.g.. non-compliant) were not detected due to insufficient sampling method or sample size.

    • Others?

    Auditing concepts techniques7

    Auditing Concepts & Techniques

    Understanding the Entity and Its Environment and Assessing the Risks of Material Misstatement (Compliance Environment)

    • Auditor should obtain an understanding of the entity’s internal compliance environment using the NERC internal compliance survey as the framework work; the framework includes:

      • Control environment (compliance)

      • Risk assessment

      • Information and communication systems

      • Control activities

      • Monitoring

        Auditor should send an internal compliance survey out before the audit to ascertain the relative risk; this is normally done in other audits which use a risk based approach

    Elements of a good internal compliance program

    Elements of A Good Internal Compliance Program

    • Requires self assessments and/or self auditing

    • Encourages self-reporting

    • Directs aggressive, timely corrective actions

    • Provides documentation quickly

    • Demonstrates knowledge of the requirements of each applicable Reliability Standard

    • Maintains organization chart for internal compliance including senior management roles

    • Demonstrates independence from operations – a direct chain of command to senior management/CEO

    • Establishes internal compliance training program

    • Disciplinary procedures for deviation from compliance

    • Controls to prevent reoccurrence of violation

    • Whistleblower policies

    • Strong transparency of process and event facts

    • A strong internal compliance program is an important mitigating factor in any enforcement action as well as helping assessing risk (refer to handout for example of survey)

    Basic auditing concepts

    Basic Auditing Concepts

    • The three types of audit risk are:




    • Key principles of Chapter 2 of GAO:

    Basic auditing concepts1

    Basic Auditing Concepts

    • Which is better audit evidence?

      • You read the policies and procedures manual of the registrant regarding their procedures in place to comply with PRC-005-01___?

      • The Compliance Manager of the registrant brags in an interview that the registrant’s procedures are so sound and controls so strong that “we can’t have an instance of non-compliance in this area and I’ll bet you lunch on it” ____?

    Auditing concepts techniques8

    Auditing Concepts & Techniques

    • Questions?

    • Follow-up items

    Module 2

    Module 2

    Audit Process and Planning

    Preparing monitoring your audit

    Preparing & Monitoring Your Audit

    • What is the “ideal” time allocation on an audit?

      Audit Planning – 10% - 20%


      • Determine applicable Reliability Standards

      • Prepare budget and staffing

      • Information request development and issuance

      • Scheduling with Registrant

      • Travel arrangements

      • Determining materiality and areas of risk

      • Determining sample sizes

      • Follow-up on findings

      • Preliminary Registrant meetings

      • Review past history – past reports, alleged violations, enforcement actions, mitigation plans, etc.

    Audit planning and supervision

    Audit Planning and Supervision

    • Planning the project – the Standard

      Planning must be documented

      • Professional judgment should be used

      • Work plans should be established

      • Needs of potential users of the reports should be considered (regulators, the registrant, industry, public interests)

      • Auditor should understand what they are auditing

      • Controls around the area to be audited should be documented and understood

      • Procedures used should be specifically designed to test compliance and to detect non-compliance

      • Criteria needed to evaluate findings should be documented in the planning stage

    Audit planning and supervision1

    Audit Planning and Supervision

    • Planning the project – the Standard

      Planning must be documented

      • Previous audits and findings should be used to focus the work plan

      • Data needed should be identified and requested through an information request from its source

      • Use of other auditor work or specialists should be considered and impact of that evaluated

      • Staffing should be sufficient to get the job done

      • Management should be contacted about audit planning – work plan and audit strategy should NOT be discussed in great detail

      • Most communication with management should be in writing

      • Professional judgment is key

    Audit planning and supervision2

    Audit Planning and Supervision

    • Planning the audit – the Standard

      Planning should be documented

      Staff should be assigned that:

      • Know the work they are doing

      • Know the subject matter

      • Have the appropriate communication skills

      • Meet the appropriate continuing education requirements

        Staffing should be assigned

      • Staff should have the proper skills or “collective” knowledge base for the job

      • Assign enough staff to get the job done

      • Provide for on-the-job staff training

      • Bring in specialists when needed

    Audit planning and supervision3

    Audit Planning and Supervision

    • Planning the audit

      Pre-audit procedures

      Organize data requests (e.g. via audit letter at least 60 days in advance)

      Cross reference information to authoritative documents (Approved Reliability Standards)

      Sufficient time to review documentation prior to field (site) work

    Audit planning and supervision4

    Audit Planning and Supervision

    This year’s experience becomes part of next year’s plan

    • Previous audits and findings should be used to shape the audit work plan and identify risks

    • Review pending mitigation plans

    • Must document that you properly planned the audit

    Electric Reliability Organization

    Regional Entity Management Group

    Prepare a project risk assessment

    Prepare a Project Risk Assessment

    • Entity will not provide sufficient evidence

    • System event disrupts availability of subject matter experts

    • _____________________

    • _____________________

    • _____________________

    • _____________________

    Checklist of planning documentation minimum

    Checklist of Planning Documentation (Minimum)

    Review Reliability Standards and requirements applicable to the registrant

    Review of how the registrant is registered (what functions?)

    Any significant changes in operations (mergers, acquisitions, etc.)

    Review of past compliance records, readiness evaluations, compliance reports, investigations and violations history

    Mitigation plans

    Surveys of other bordering registrants

    Compliance audit letter

    List of primary contacts

    Logistics for the audit

    Estimates of audit hours

    List of external, outsourced contractors and their purpose

    Documentation related to meeting audit team conflicts and independence requirements

    Identification of audit team members and their roles on audit

    Documentation that audit team reviewed current CMEP

    Internal compliance survey

    Staffing the audit

    Staffing the Audit

    • What is leverage and how can it help?

      Leverage - Assigning audit activities based on level of experience

      • Goals:

        • Raise experience level of staff

        • Get project done in most efficient manner

        • Maximize resources, i.e. focus on high priority tasks for staff

        • Assign most efficient use of outsourcing resources

        • Meet annual audit scheduling needs

        • Help staff meet hours goals

    Staffing the audit1

    Staffing the Audit

    • What is a good leverage model?

      Experienced auditors

      • Research RE and get to know their business activities and relevant audit areas

      • Serve as RE contact

      • Audit planning and staffing determination

      • Design audit tests

      • Lead audit fieldwork and assign tasks

      • Conduct exit interview with RE

      • Review audit work papers – assign follow-up work to audit senior to clear items

      • Prepare high level sections of report

      • Review other areas of report

      • Be responsible for report content and findings

      • Prepare staff evaluations

      • Prepare expert witness testimony

      • Serve as expert witness

      • Train staff in leading smaller audits

    Staffing the audit2

    Staffing the Audit

    • What is a good leverage model?

      Senior auditors

      • Research registrant and get to know their business activities and relevant audit areas

      • Serve as registrant contact

      • Assist with audit planning and staffing determination

      • Assist in designing audit tests

      • Supervise audit staff in their fieldwork activities

      • Participate in exit interview with registrant – oversee follow-up questions

      • Complete audit work papers and review staff work papers – assign follow-up work to audit staff to clear items

      • Complete audit work papers and checklists

      • Prepare detailed sections of report and review other areas of report

      • Train and prepare staff evaluations

      • Assist in preparing expert witness testimony (if needed)

      • Gain experience in supervising and leading larger audits

    Staffing the audit3

    Staffing the Audit

    • What is a good leverage model?

      Staff auditors

      • Research registrant and get to know their business activities and relevant audit areas

      • Assist in audit planning

      • Performed detailed audit tests and sampling

      • Propose audit findings to audit leader

      • Attend exit interview with registrant and assist with questions

      • Complete audit work papers and checklists

      • Clear review comments on work papers as assigned

      • Assist in preparation of expert witness testimony (if needed)

      • Participate in evaluation meeting with audit leader or audit senior

      • Gain experience to become audit senior

    Audit planning

    Audit Planning

    • Planning the audit

      Use of a Specialist

      • You should be able to tell the specialist what needs to be done

      • You need to be able to identify whether what the specialist does meets your needs

      • You need to be able to evaluate the specialist’s results to apply to other areas of the audit

      • External specialists must be under the same independence and conflict rules of staff

    Audit planning1

    Audit Planning

    • Some benefits:

    • A properly planned audit mitigates risk

      • Risk of failure includes losing certification to perform compliance audits, reputational risk

      • Proper planning will translate into fewer challenges in the findings

    • Proper planning demonstrates professionalism and competency

    Preparing monitoring your audit1

    Preparing & Monitoring Your Audit

    • What is the “ideal” time allocation on an audit?

      Audit Performance (Fieldwork, off and on site) – 60% - 70%


      • Pre-audit staff review of registrant information and evidence

      • Testing

      • Prepare detailed work papers to demonstrate findings

      • Discuss preliminary findings and information to clear them if needed

      • Closing conference/briefing with Registrant including Audit results

      • Identify open item list and responsibilities for delivery with Registered Entity

      • Write draft report

    Preparing monitoring your audit2

    Preparing & Monitoring Your Audit

    • What is the “ideal” time allocation on an audit?

      Audit Performance (Fieldwork) – 60% - 70% (focus on the value)

      • Design testing to gain adequate coverage for “high risk” areas

      • Leave the little stuff alone – use 80/20 rule

      • Define the goal of testing in each area and balance between findings and providing recommendations for solutions to registrant

      • Use fieldwork as the opportunity to train staff

      • Always get as much as possible done in the field – resolve open issues

      • If an alleged violation is discovered, auditor should determine: Is it a risk to bulk power system reliability?

    Audit performance

    Audit Performance

    Staff Supervision - Practice the W approach

    • Why are we doing this?

    • What are we supposed to get done?

    • Please show me how to do it….

    Audit performance1

    Audit Performance

    • Staff are to be properly supervised

      • Ensure the audit objectives are accomplished

      • Provide guidance

      • Stay informed about problems encountered

      • Review the work

      • Train the staff

    Preparing monitoring your audit3

    Preparing & Monitoring Your Audit

    • What is the “ideal” time allocation on an audit?

      Audit Reporting – 30% or less


      • Clear open item lists

      • Complete work papers

      • Prepare draft report

      • Issue draft for internal review

      • Comply with internal and CMEP requirements

      • Issue draft report to registrant for their review and comment

      • Issue final report

      • Make notes for next audit of the registrant

      • Prepare staff evaluations

    Due professional care judgment

    Due Professional Care & Judgment

    • Auditor must plan and perform audit to obtain appropriate evidence so that audit risk is limited to a low level appropriate for expressing a conclusion or position

    Audit process

    Audit Process

    • Lunch provided on-premises by the registrant to the audit team is permissible. However, you can not go to a restaurant with the registrant and they pick up the tab

    • True False

    • 2. The best way to efficiently complete the audit is for the audit lead to do most of the hands-on testing and interviews

    • True False

    Audit process1

    Audit Process

    • 3.Audit planning requires the audit lead to complete all checklists and information requests once at the registrant site and then wait for information to be provided:

    • True False

    • The registrant invites the audit team to join them for happy hour to discuss how the audit is going – does this impair independence of the team?

    • True False

    Audit process2

    Audit Process

    • One of the registrant’s staffers just had a new addition to the family. One of the audit team members chips in for a gift. This:


      does not impair independence.

    • The registrant’s staff seem like “good people”. One of the registrant says “don’t’ worry about testing PRC-005-01, we’re all over it”. The audit team is running short of time in the field and seeking for areas of minimal risk to pass testing on. PRC-005-01 is an area that

      should be

      should not be on the “to be rotated list” for registrant self-reporting.

    Audit process and planning

    Audit Process and Planning

    • Questions

    • Section review

    • Follow-up items

    Module 3

    Module 3

    Managing Relations

    Electric Reliability Organization

    Regional Entity Management Group

    Learning objectives1

    Learning Objectives

    • Learn keys to establishing good relations

    • Balance trust with audit responsibilities

    • Empathy and active listening

    • What to do when relationships are strained

    Electric Reliability Organization

    Regional Entity Management Group

    Audit principles

    What are the concerns of your registrant?1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

    Electric Reliability Organization

    Regional Entity Management Group

    What are the concerns of your registrant

    What are the concerns of your registrant?

    • Possible responses:

    • What are they really looking for?

    • How will this work?

    • How am I going to get all of this done and keep my ‘real job’ going?

    • Do they have the knowledge and experience to understand what we do?

    • If they find something, what will happen?

    • Will they keep my confidential information safe?

    • Perhaps if I go slowly, they will go away?

    • Will we receive a large fine?

    Electric Reliability Organization

    Regional Entity Management Group

    How to establish good relations

    How to establish good relations

    • Establish trust – use the Golden Rule

    • Suspend judgment

    • Communicate, communicate, communicate

    • Stick to the plan

    • Tell them what you are going to do, tell them what you are doing, tell them what you did

    • Use empathic responses

    • Practice active listening

    • Know your role

    • Communicate, communicate, communicate

    Electric Reliability Organization

    Regional Entity Management Group

    How to establish trust

    How to establish trust

    • Credibility builds trust

      • Knowledge

      • Skill

      • Competency

    • Share information

    • Be fair and honest

    • Meet your commitments

    • Listen well

    • Perform competently

    • Be predictable

    • Communicate openly and clearly

    Trust is essential for conflict resolution – trust is a key component as it is associated with enhanced cooperation, information sharing and problem solving

    Electric Reliability Organization

    Regional Entity Management Group

    Use empathic responses

    Use Empathic Responses

    • Empathy is often described as “putting oneself in another’s shoes” or…

    • “the ability to put oneself in the role of another”.

    • Empathy is not pity nor sympathy, but instead the ability to understand what it feels like to be the subject of an audit.

    Electric Reliability Organization

    Regional Entity Management Group

    Active listening

    Active Listening

    • Active listening is an intent to listen for meaning.

    • The listener checks with the speaker to see that a statement has been correctly heard and understood. The goal of active listening is to improve mutual understanding

    • Watch body language

    • Paraphrase the speaker’s words

    • Let the speaker know that you are hearing what they are saying

    • Give the speaker your full attention – if you are doing other things, they will feel not heard.

    • Nod occasionally, smile and react

    • Encourage with small verbal comments (yes, oh, ummm…)

    • Make your posture open and inviting

    • Ask clarifying questions

    • Summarize what you have heard

    Electric Reliability Organization

    Regional Entity Management Group

    What should you do if relations seem strained

    What should you do if relations seem strained

    • Raise any concerns first with your team leader or manager

    • Do not assume that the root cause is your behavior – it may be other stresses in the auditee’s world – we all have a bad day

    • Proactively observe interactions and seek assistance

      • Involving other team members sometimes can help

      • Sometimes the “chemistry” between individuals just doesn’t work – don’t let it fester, seek help from your team leader

      • Be professional and objective; if there are genuine disagreements on “findings”; listen, consider, but, at the end of the day the auditor’s judgement and determination must be relied upon in the public domain; politely remind the registrant of due process protections through the enforcement processes (point to the CMEP)

      • If registrant feels auditors are acting unfairly, request registrant contact senior executive at Regional Entity; registrants do have a right to disagree; audit team members should not engage in “debates”; again, there are formal due process protections when there are “findings” (e.g. potential alleged violation)

    Electric Reliability Organization

    Regional Entity Management Group

    Managing relations

    Managing Relations

    • You and the compliance manager of the registrant have not hit it off. You are scheduled to interview the compliance manager in connection with the field audit. You should

    • ___ Continue with the audit as planned

    • ___ Add another of the audit staff to the interview for support and note taking

    • ___ Cancel the interview and schedule someone else

    • ___ Ask another audit staff to do the interview

    Electric Reliability Organization

    Regional Entity Management Group

    Managing relations1

    Managing Relations

    • 2. The registrant’s compliance manager has said to your boss that the report filed is a “bunch of bs and full of misstatements!” You should:

    • ____ Revise the report based on the compliance manager’s comments

    • ____ Add another finding to the report on the compliance manager’s lack of agreement with your report

    • ____ Point out the evidence found supporting your assertions to the compliance manager in the exit conference

    Electric Reliability Organization

    Regional Entity Management Group

    Managing relations2

    Managing Relations

    • Questions

    • Section review

    • Follow-up items

    Electric Reliability Organization

    Regional Entity Management Group

    Module 4

    Module 4

    Audit Documentation & Evidence

    Learning objectives2

    Learning Objectives

    • What do the audit standards require for documentation?

    • What’s the “yellow book”?

    • What do you need to do to meet GAGAS evidence requirements?

    • What types of audit work papers are there?

    • What is “quality” when it comes to audit tests?

    Audit documentation and evidence

    Audit Documentation and Evidence

    • Auditors should prepare and maintain audit documentation. Audit documentation related to planning, conducting, and reporting on the audit should contain sufficient information to enable an experienced auditor, who has had no previous connection with the audit, to ascertain from the audit documentation the evidence that supports the auditors’ significant judgments and conclusions. Audit documentation should contain support for findings, conclusions, and recommendations before auditors issue their report.

    Audit documentation and evidence1

    Audit Documentation and Evidence

    • Audit Documentation Standards

    • The form and content of audit documentation should be designed to meet the circumstances of the particular audit. The information contained in audit documentation constitutes the principal record of the work that the auditors have performed in accordance with standards and the conclusions that the auditors have reached. The quantity, type, and content of audit documentation are a matter of the auditors’ professional judgment.

    Audit documentation and evidence2

    Audit Documentation and Evidence

    • Audit Documentation Standards

    • Audit documentation serves to:

      • Provide the principal support for the auditor’s report

      • Aid in conducting and supervising the audit

      • Allow for the review of audit quality

    Audit documentation and evidence3

    Audit Documentation and Evidence

    • Audit Documentation Standards

      Documentation should provide these items:

      • Objectives, scope and methodology of the audit, including sampling and other selection criteria

      • Auditors’ determination that certain standards do not apply or that an applicable standard was not followed, the reasons therefore, and the known effect that not following the applicable standard had, or could have had, on the audit

      • Work performed to support significant judgments and conclusions (including descriptions of procedures and records examined)

      • Evidence of supervisory reviews before the report is issued

    Audit documentation and evidence4

    Audit Documentation and Evidence

    • Evidence and documentation in an audit must provide the following support to determine whether a registrant is compliant with a standard/requirement:

      • Existence or occurrence –recorded events or activities have occurred

      • Completeness –everything that happened is presented and events or activities presented are in accordance with industry standards

      • Accuracy and classification –events and activities presented are accurate and correctly presented

    Audit documentation and evidence5

    Audit Documentation and Evidence

    • Basic GAO or “Yellow Book” procedures

      • Auditor mustidentifypertinent regulations (CMEP and NERC)

      • Auditor mustassessthe risks of materials non-compliance

        • Assess internal controls (compliance environment)

    • Auditor must design steps and procedures totestcompliance with regulations to ensure that both unintentional and intentional instances of material noncompliance are detected

    • Auditorissuesreport on tests of compliance in which all instances of noncompliance or violations must be reported

    Audit documentation and evidence6

    Audit Documentation and Evidence

    • Audit Evidence

      • Any information that corroborates or refutes an assertion of compliance

    • Objective of Audit Documentation

      • Provide principal support for the representations in an auditor’s report

      • Assist in planning, performance and supervision of the engagement

      • Extraneous (un-needed) audit evidence not needed should not be included in work papers

    Audit documentation and evidence7

    Audit Documentation and Evidence

    • “Sufficiency of Audit Documentation”

      • Enables members of the audit team with supervision and review responsibilities to understand the nature, timing and results of auditing procedures performed and evidence obtained

      • Indicates which audit team members did the work

      • Shows that the records examined agree with the assertions being tested

      • Acid test – if you have no connection with the audit, can you come to the same conclusions? Uninformed Reviewer Test

    Audit documentation and evidence8

    Audit Documentation and Evidence

    • “Sufficiency of Audit Documentation”

      • All audit documentation should be complete before the report is issued

      • Any post-audit procedures should be dated and identified

      • Lockout report and findings 60 days after completion – no modifications after lockout; limiting access via lockout prevents potential manipulation of findings and reports

      • Audit document must support compliance or create the record for non-compliance; evidence must stand on its own merits

    Audit documentation and evidence9

    Audit Documentation and Evidence

    • Retention of Audit Work Papers

      • General rules are five years

      • Follow reliability standards, if greater than five years or minimum audit cycle (whichever is longer)

      • Retain until controversy is resolved

    • Standards require adequately safeguarding audit documentation—make them secure!!

    • Standards require defined policies for release of audit documentation to outside parties in keeping with laws and regulations that apply to both the audited registrant and audit organization

    Audit documentation and evidence10

    Audit Documentation and Evidence

    • Quality of Audit Evidence

    • Influenced by its Source and Nature

      • Knowledgeable independent sources

      • Generated internally when controls are effective

      • Directly obtained evidence by the auditor (observation) vs. inquiry

      • Documentation of events (i.e.. written logs vs. oral representation)

      • Original documents vs. reproduction (copies and fax)

      • SCADA/EMS screenshots

      • Emails or other electronic documentation

      • Operating procedures

      • Phone transcripts

    Audit documentation and evidence11

    Audit Documentation and Evidence

    • Quality of Audit Evidence

    • Some evidence is better than others

      • Evidence obtained when internal compliance program and related controls are effective is better than when internal controls are not effective

      • Direct evidence (physical examination, observation, logs, records, computation and inspection) is better than indirect evidence (procedures)

      • Original documents are better than copies

      • Testimonial evidence is more reliable if it is given freely

      • Testimonial evidence from unbiased party is better

      • Evidence from credible third party may be more competent than that of management

    Audit documentation and evidence12

    Audit Documentation and Evidence

    • Quality of Audit Evidence

    • Some evidence is better than others

      • Written representations from management confirm oral representations

        • Certifications from officers are required given the nature of the auditors work (similar to Sarbanes Oxley)

        • Representations can be upfront via a general certification or may be done during the audit on an as-needed basis

        • Can be used when other documentation is not available (can be in the form of an affidavit, if it’s a material matter)

      • Data can be gathered by auditors through their own observations and measurements (interviews, questionnaires, observation, computations)

    Audit documentation and evidence13

    Audit Documentation and Evidence

    • Quality of Audit Evidence

    • Some evidence is better than others

      • Data can be gathered by management– auditor must determine if information is valid and reliable

      • Third parties can gather data – auditor must determine reliance on that data

      • Corroborate weaker evidence with more evidence to gain reasonable assurance

      • Should not base evidence quality or other decisions in the audit upon a potential monetary penalty. Suggest auditors separate a “finding” from the enforcement implications to keep it “clean”.

    Regional Entity Management Group

    Audit documentation and evidence14

    Audit Documentation and Evidence

    • Audit Procedures for Obtaining Audit Evidence

      • Inspection of records or documents

      • Inspection of tangible assets

      • Inquiry

      • Confirmation

      • Recalculation

      • Re-performance

      • Analytical procedures

        ** Breakout Exercise

    Need for quality in gathering and supporting audit evidence

    Need for Quality in Gathering and Supporting Audit Evidence

    • Quality of Report is Supported only by the Evidence Gathered

    Gathering quality evidence

    Gathering Quality Evidence

    Know the RSAW

    Know the standard you’re testing

    What is the linkage?

    Follow the RSAW testing requirements line by line

    There MUST be evidence that supports every line item tested

    What did you do? How did you do it? What piece of evidence did you select and why?


    Gathering quality evidence1

    Gathering Quality Evidence

    If you can’t explain why you selected a piece of evidence and how it supports your position, it isn’t good evidence – discard it (it will cause you problems later)

    Every piece of evidence must have a direct link to the report or it should not be in the file

    Conversely – there should be no unsupported statements in the report


    Writing a quality report

    Writing a Quality Report

    Reports should include:

    A clear explanation of the requirements in terms that are understandable to the general public

    A description of the method of testing compliance, including testing method used, population definition, sample size and results found

    A record of interviews (date, time, participants, subjects covered) with direct quotes on pertinent topics

    A description of audit results which demonstrate non-compliance with standards and meet standard of legal sufficiency

    Explanation/argument which help the reader understand that the results (evidence) found equate to non-compliance

    Documentation of results review with registrant

    Audit documentation and evidence15

    Audit Documentation and Evidence

    • Main tool in auditing is “sampling”

      • Sampling is… a systematic and defensible approach to drawing a conclusion of a population based on reviewing less than 100% of that population

    • Why not sample 100% of the transactions?:

      • Lack of time and resources

      • How do you know you have 100% of the population?

      • Finding “one more” negates validity of the entire sample

      • Rely on more on systematic controls in place in some cases for reliance that compliance is being met

    Audit documentation and evidence16

    Audit Documentation and Evidence

    • Sampling Methods




    Population Proportional


    Audit documentation and evidence17

    Audit Documentation and Evidence

    • What are the main tools used in compiling audit documentation?

      • Work programs (RSAWs)

      • Checklists

      • Digital files

      • Audit extraction tools

      • Manual work papers

      • Representation letters

      • Formal reports

    Audit documentation and evidence18

    Audit Documentation and Evidence

    • Suggested File Construction (NERC and RE’s should establish a standard file construction)

      • Current files

        • Applicable standards and RSAW’s

        • Active mitigation plans

        • Pending violations

        • Internal compliance survey

        • Current self certifications

      • Permanent files

        • Documents overall registrant information needed for ongoing audits

        • Organization structure, corporate information

      • Archived files

        • Past audits and other materials

          • Findings communicated to management

          • Management responses

        • Self certifications

        • Completed mitigation plans

    Audit documentation and evidence19

    Audit Documentation and Evidence

    • The yellow book is:

    • ____ A book where you can find good restaurants to visit after field work is completed for the day

    • ____ A guidebook from the GAO on audit standards

    • Audit Evidence is:

      ____ Any information that corroborates or refutes an assertion of compliance

      ____ Documentation that will be admissible in hearings for non-compliance

      ____ All of the above

    Audit documentation and evidence20

    Audit Documentation and Evidence

    • 3.The objective of Audit Documentation is to:

      ____ Provide principal support for the representations in an auditor’s report

      ____ Assist in planning, performance and supervision of the engagement

      ____ All of the above

    • Lack of audit evidence will lead to:

    • ____ Embarrassment on the witness stand should you testify in hearings as to the assertions in your report

    • ____ Reversal of fines assessed for non-compliance by the RE

    • ____ All of the above

    Audit documentation and evidence21

    Audit Documentation and Evidence

    • Lack of audit evidence in support of report assertions by your organization may lead to what types of actions?

    • _____ Reversal of fines assessed for non- compliance by the RE

    • _____ Revocation of the right to perform compliance audits by your organization

    • _____ All of the above

    Audit documentation and evidence22

    Audit Documentation and Evidence

    • Questions?

    • Follow-up items



    Compliance Monitoring and Enforcement Program Auditor Training

    RSAW Workout

    Audit principles

    Compliance Audits

    • The following example is from an audit, the names and references have been changed.

    • Also following are tools and excerpts to help with the workout.

    Audit principles

    Audit Questionnaire

    Audit principles

    Audit Phone Trans-cripts


    one transcripts for Midwest Electric Co.



    002 R2, February 10, 2008




    Step number 1


    Uh, for the Fishie sub finished up 115kV bkr 5201 time 0806

    Now you are ready for us to do this other part


    Aren’t you?


    Give me a call back

    Are you in the clear?

    I am in the clear





    This is Josh

    Line 3340 is out of service



    Alright, thanks Jim


    That would be clearance order 7901, give you a


    Jesus yeah, 7



    ME this is Josh

    Hey Josh, this is Donny, we are going to start switching on 115 line number 7577a at


    Alrighty, I will put her down


    Audit principles

    Audit Phone Trans-cripts


    This is Josh


    Yes, Josh uh, circuit 6742 at 40

    and Lea is on non reclose and tagged for your





    Control this is Josh

    Hey Josh, this is Bob


    Order 7086

    Right on

    Ok, step 5 install grounds at Treat Junction between both open points


    Got all that

    Got all that


    Audit example

    Audit Example

    • Questions?

    • Section review and quiz

    • Follow-up items

    Module 5

    Module 5

    Work Paper Preparation and Information Requests

    Learning objectives3

    Learning Objectives

    • What are the audit documentation standards?

    • What are the building blocks for work paper preparation?

    • How to prepare an effective information request

    • When to use third party confirmations

    • What is a “Compliance Audit or Get Ready” letter

    • How to track requests

    Work paper preparation

    Work Paper Preparation

    • Audit Documentation Standards

    • Key characteristics

      • Sufficient and complete

      • Written

      • Support findings (compliance or non-compliance)

      • Professional judgment - quantity, type and content

    • Audit work papers are the principal record of the work performed by the auditor; work papers form the foundation for findings of compliance

    Work paper preparation1

    Work Paper Preparation

    • Common Paperless Audit Work Paper Platforms

    • Word

    • Excel

    • Visio

    • Access

    • Adobe

    • TIF files

    • Data extraction tools

      • Does any Regional Entity staff have a data extraction tool? Are these necessary for a compliance audit?

    Work paper preparation2

    Work Paper Preparation

    • Common Paperless Audit Work Paper Platforms

    • Audit standard setting bodies including GAO allow use of electronic work papers (in fact, electronic formats are essential to manage the volume of information)

    • Court cases allow submittal of electronic work papers into evidence

    • Regulatory bodies regularly use electronic work papers in hearings and proceedings

    Work paper preparation3

    Work Paper Preparation

    • Use References, if necessary-Common References

    • CConfirmed

    • TTraced to Records (indicate type)

    • NA Not applicable

    • NN Not Considered Necessary

    • PY Ties to Prior Year's Working Papers

    • VVouched

    • RRecalculated

    • CFCross footed

    • FFooted

    • PIPhysically Inspected

    • PRequested Positive Confirm

    • NRequested Negative Confirm

    • PFTies to Permanent File

    • NW No Further Work Necessary

    • PV Possible Violation

    Confirmations and their use

    Confirmations and their Use

    • Positive Confirmation

      • Used when confirming evidence purported by a registrant as being true

      • Requires action by the confirming party

      • Very reliable evidence

      • Follow-up needed for non-response by confirmee

      • Example-use to confirm that a proper action was performed by a BA from the RC; or vice versa

      • Positive confirmations are prepared by the auditor on their letterhead

      • Use judgment. Positive confirmation are useful to corroborate weak evidence

    • Negative Confirmation

      • Used when confirming evidence purported by the registrant as being true

      • Requires NO action by the confirming party

      • Less reliable evidence than positive confirmations

      • Follow-up needed for response that indicates disagreement by the confirming party with the evidence to be confirmed

    Work paper preparation4

    Work Paper Preparation

    Positive Confirmation (on auditor letterhead)

    • Date

    • Party to be confirmed

    • Dear xxxx:

    • Regional Entity, XXXX, are performing a compliance audit under section 215 of the Federal Power Act; as such Regional Entity is examining certain required communications and/or actions involved between Registrant and Confirmed Party. In connection with this examination, please confirm directly the correctness of the information shown below:

    • INFORMATION TO BE CONFIRMED (also reference standard and requirement)

    • A business reply envelope is enclosed for your convenience.

    • Sincerely,

    • XXXX Auditor

    • >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    • The information contained in this letter is correct to the best of our knowledge.

    • Signed – (Confirmed party representative)

    Work paper preparation5

    Work Paper Preparation

    Confirmation Tracking

    Audit principles

    Compliance Audit Letter

    • One of your first contacts with a registered entity is at least 60 days via a notice.

    • Information request is built off of your audit planning phase. It is intended to ask for pertinent data which will provide the total population from which you will draw your samples.

    • Often this request is included by a “Compliance Audit or Get Ready Letter”

    • NERC and Regional Entities have standardized the letter (refer to handouts)

    Compliance audit letter

    Compliance Audit Letter

    • The key elements of Compliance Audit Letter are…

      • Includes certain representations and certifications (see examples-letter to Corporate Officer; letter to Primary Compliance Contact)

      • Advanced notice and reminder that it is time to get ready for the audit

      • Confirmation of scheduled audit dates

      • How you expect to receive the requested information (hard copies, electronic files, etc.)

      • An attached Audit Questionnaire (checklist or information request document) which can be used to organize the information and record transmittal

      • Requested due dates

      • Audit team members (with or without bio’s)

    Information requests guidelines

    Information Requests Guidelines

    • Often it takes significant effort to the registered entity to collect and organize all of the information necessary for you to perform your sampling and testing.

    • All requests should be in writing and include:

      • Clear description of information requested

      • Date information should be received

      • Date information is received

      • Information request number with each line of requested information clearly identified with a unique number

    • All information requests should be logged initially and receipt of information recorded on the log.

    Clearly request information

    Clearly Request Information

    • Your audit team should use a standard template for all information requests for additional information requested beyond the questionnaire.

    Information request log

    Information Request Log

    • All data requested should be logged to track and protect registrant information. The following template shows the minimum information that should be recorded.

    Work paper preparation standards

    Work Paper Preparation Standards

    • The notation “PV” means “physically inspected”

    • True False

    • Positive confirmations are always more reliable than negative confirmations because they require a member of the audit team to visit the confirming party’s location and interview them before we can confirm the quality of the evidence.

    • True False

    • ____ days prior to audit field work information should be requested from registrant. Can this info be requested via phone call or email?

    • True False

    • Work papers can be in an electronic format to be admissible in hearings

    • True False

    Work paper preparation and information requests

    Work Paper Preparation and Information Requests

    • Questions

    • Section review

    • Follow-up items

    Module 6

    Module 6

    Audit Testing and Testing Methodologies

    Learning objectives4

    Learning Objectives

    • What are the different kinds of sampling?

    • How do we do it?

    • Why sample?

    Audit testing and testing methodologies

    Audit Testing and Testing Methodologies

    • Main tool in auditing is “sampling”

    • What is sampling?

    • Sampling is a systematic and defensible approach to drawing a conclusion of a population based on reviewing less than 100% of that population

    Audit testing and testing methodologies1

    Audit Testing and Testing Methodologies

    Why not sample 100% of the transactions?

    • Lack of time and resources

    • How do you know you have 100% of the population?

    • Finding “one more” negates validity of the entire sample

    • Rely on more on systematic controls in place in some cases for reliance that compliance is being met

    Sampling definitions

    Sampling Definitions

    • Deviation Rate/Error Rate

      • Rate of acceptable errors in sample

    • Tolerable/Confidence Level

      • Rate of confidence that the auditor has that the sample selected is representative of the entire population

    • Population

      • Total number of items that can be tested

    Audit testing and testing methodologies2

    Audit Testing and Testing Methodologies

    Error Rate We’re Willing to Accept

    Audit testing and testing methodologies3

    Audit Testing and Testing Methodologies

    Generally accepted statistical sampling methods (SAS 111 – Amendment to Statement on Auditing Standards No. 39)

    • Haphazard

    • Systematic

    • Random

    Audit testing and testing methodologies4

    Audit Testing and Testing Methodologies

    Statistical sampling

    • The magic number is 45 for no errors at a 5% tolerable rate

    • The magic number is 77 for one error (helps determine the severity of the violation)

    • After that, stop the sample!

    Audit testing and testing methodologies5

    Audit Testing and Testing Methodologies

    • Haphazard

      • Reach and grab approach; may be appropriate in certain situations

    Audit testing and testing methodologies6

    Audit Testing and Testing Methodologies

    • Systematic

      • Select every nth item

      • Should be used in conjunction with statistical sampling

    Audit testing and testing methodologies7

    Audit Testing and Testing Methodologies

    • Random

      • If items in population are identifiable by a number scheme, use Excel or other random number generation tool to determine items to be sampled

    Audit testing and testing methodologies8

    Audit Testing and Testing Methodologies

    Sample of Attributes Testing Work Paper

    Audit testing and testing methodologies9

    Audit Testing and Testing Methodologies

    Other Testing Methods

    The 65% rule

    The 100% rule

    The “Test of One”

    Testing to an insignificant level of remaining population, i.e.. 35% of the population

    Expanding sample when an error is encountered

    Stopping testing when an error is encountered – One and Done!

    Audit testing and testing methodologies10

    Audit Testing and Testing Methodologies

    What do you do when you find an error (potential violation in your sample)?

    Increase sample size to the next level, if no additional errors (potential violations) = STOP!

    If additional violations, consult enforcement staff….think about….

    Requesting registrant to perform a 100% analysis or a larger subset of information to determine severity or pervasiveness of errors (potential violations)

    Increasing the VSL of enforcement action (no additional sampling, document sampling result and include in work papers (i.e. record) for enforcement; additional testing by registrant can be included as part of a mitigation plan).

    Example-one error (potential violation) found in sample of 65 is much different than finding four errors (potential violations) in the same sample size

    NERC and Regional Entities should establish standardized procedures regarding sampling

    Audit testing and testing methodologies11

    Audit Testing and Testing Methodologies


    ….thequalityof yourtestis dependent on the quality of the information you get for testing

    ….thedepthof yourtestis dependent on the depth of your information request

    ….thequalityof theevidenceyou gather is dependent on the cooperation you receive from the RE and their staff

    …..thequalityof yourfindingsand recommendations are totally dependent on all of the above

    Audit testing and testing methodologies12

    Audit Testing and Testing Methodologies

    • The main kinds of statistical sampling are:





    • The main advantage to sampling is:

    • ____________________________________________________________________________________________________________________________________

    Audit testing and testing methodologies13

    Audit Testing and Testing Methodologies

    • 3. The initial sample size is always ___ items at a 95% confidence level

    • 4. If a piece of evidence is sampled that does not meet the assertion (standard or requirement) being tested we will sample another ____ items.

    • 5. If another piece of evidence is found in the sample that does not meet the assertion being tested what do we do?

    • _________________________________

    • It is never acceptable to have a sample size of one item of the population

    • True False

    Audit testing and testing methodologies14

    Audit Testing and Testing Methodologies

    • Questions?

    • Follow-up items

    Module 7

    Module 7

    Interview Techniques

    Learning objectives5

    Learning Objectives

    • Techniques and tools used in the interview process

    • Standardized interview questions

    • Roles in the interview process

    • Meeting notices, interview records and logs

    Why conduct interviews

    Why Conduct Interviews?

    • It’s a productive way of obtaining information

    • Opportunity to obtain information not found in documentation

    • Obtain off-record information

    • Meet knowledgeable & involved people and help us understand the organization being audited

    • Conduct interviews to supplement and/or clarify documentation

    • Verify procedures are being followed

    How to interview effectively

    How to Interview Effectively

    • Prepare list of questions in advance.

    • Start with introductory questions (name, title, organization, background)

    • Ask questions related to interviewee’s competence or area of expertise

      • A well prepared set of interview questions ensures your confidence in the completeness of information obtained efficiently and effectively

    Preparing for the interview

    Preparing for the Interview

    • Identify need for interview and appropriate registered entity personnel

    • Call and set up interview. Be ready to describe the purpose of the interview

    • Prepare and send Interview Request form

    • Log the scheduled interview, date and purpose

    • Prepare written questions to ensure a complete and logical agenda for the meeting

    • Assign roles to attending audit team members

    • For ad hoc requested interviews, follow up with formal interview request

    Sample interview request form

    Sample Interview Request Form

    Conducting the interview

    Conducting the Interview

    • 1.Share the purpose of the audit

      • Objectives, goals, timing, confidentiality, and format

      • Audit teams’ roles and responsibilities

      • Explain what is expected from the registrant

    • 2. Use a sociable, conversational style

      • People will be more cooperative, less guarded

      • Avoid being harsh, grilling, or demanding

      • Be professional at all times

        • Avoid cursing

        • Don’t ask irrelevant questioning and discussions

    • 3.Let them know you will be taking minutes

    • 4.Use active listening

      • Polite responses

      • Ask for more….

      • Repeat back what you have heard

    Types of questions to ask

    Types of Questions to Ask








    • Advantage

    • Generates explanations and descriptions

    • “ explain to me, describe, show me…”

    • Gets simple information

    • used tocheck facts, short answers, yes/no

    • Uncovers core issues

    • “How do you know...?

    • Ask when answers received contradict previous answers or observations

    • Check your understanding of what was said

    • “So let me see if I understand correctly…”

    • Ties together the main points covered in a session

    Leading questions

    Leading Questions

    • Definition:

    • A leading question is one that suggests an answer, that implies there is a correct answer.

    • The term comes from law, where the courts insist that questions that suggest answers are not asked because they restrict the right of witnesses to speak freely.

    Use of leading questions

    Use of Leading Questions

    • A leading question is a question that suggests the answer or contains the information the auditor is looking for.

    • For example:

    • “You were at Duffy's bar on the night of July 15, weren't you?”

    • (It suggests that the person was at Duffy's bar on the night in question).

    • The same question in a non-leading form would be:

    • “Where were you on the night of July 15?”

    • (This form of question does not suggest the answer the auditor hopes to elicit).

    Leading questions1

    Leading Questions

    • Leading questions may often be answerable with a YES or NO, while non-leading questions are ‘open ended’. They also:

      • Point the respondent in the right way

      • Can be interpreted that the interviewer is not objective

    • Use of leading questions can be used where you are looking for information and the entity may use different naming criteria than what the auditor is used to. Use careful judgment in using leading questions.

    Sample interview summary record

    Sample Interview Summary Record

    More tips

    More Tips…

    • Maintain eye contact

    • Be aware of body language

    • Take notes

      • It is often good for one person to ask the questions and have a second person to act as recorder

    • Talk little

    • Avoid answer-suggesting, accusatory and judgmental questions

    • Debrief after the interview – schedule a half hour between interviews to complete interview notes, use the restroom and get ready for the next interview

    • Explain the next steps in the process

    In summary

    In Summary…

    • Set a positive environment for the interviews

    • Use a conversational style during the interview

    • Maintain control of the interview

    • Use different types of questions during the interview

    • Use a variety of techniques for handling challenging interviews

    Interview techniques

    Interview Techniques

    • Interview notes are ___ always ___ never admissible in compliance hearings.

    • Interview notes cannot ___ can ___ be used in reports in regards to findings of non-compliance.

    • Interview notes without a corroborating witness in the interview ____ are ___ are not considered audit evidence.

    Interview techniques1

    Interview Techniques

    • 4. You have scheduled an interview with the Compliance Manager of the RE. Due to scheduling issues you are not able to do the interview during audit field work. The deadline for issuance of the report is fast approaching and this interview is not yet completed. What do you do?

    • ___ Rotate the interview until the next field audit

    • ___ Issue the audit report without doing the interview

    • ___ Hold off issuing the audit report until you can complete the interview

    • ___ Consider the lack of an interview a report finding

    Interview techniques2

    Interview Techniques

    • 5. Interview requests, logs and notes should ____sometimes ___always be documented.

    • 6. It is ___ok ___not ok to rely on your memory in including interview comments in reports or hearings.

    Interview techniques3

    Interview Techniques

    • Questions

    • Section review

    • Follow-up items

    Module 8

    Module 8

    Leveraging Project Management Tools & Techniques for Audit Success

    Learning objectives6

    Learning Objectives

    • Project management audit tools and techniques

    • Why is budget important

    • Interview records and logs

    • Issue logs

    • Status reporting tools

    • Internal reporting issues log

    • Other dashboard tools

    Audit principles

    What is Project Management?





    • Project management is the process that will help us coordinate audits at a sufficient level of detail so that timely decisions can be made andrisks can be minimized.

    • This includes an incorporation of your knowledge, project management tools, skills, and techniques.

    • Projects have a beginning and an end.

    • Three phases of an audit:

    • 1. Planning

    • 2. Performance

    • 3. Reporting

    Audit principles

    Why Use a Disciplined Approach?

    • Project management will require you to focus on…

      • scope

      • cost

      • quality

      • effort

      • risk

      • timelines

    • … to ensure high quality service and audit success!

    Why do audits and projects fail

    Why do audits and projects fail?

    • Lack of follow-through and accountability

    • Lack of time and focused resources

    • Loss of momentum or focus

    • Poor planning

    • Inability to transfer skills or knowledge to team or client

    • Lack of vision

    • Lack of cohesive leadership team

    • Inadequate emphasis on speed

    • Failure to allocate necessary resources

    • Organizational confusion and division

    • Lack of decision support information to measure success

    • Failure to maintain focus on the client

    • Errors in estimating how long tasks will take and what it will cost to do them

    Audit principles

    Project Management Life Cycle

    Executing & Controlling: Managing the execution

    of tasks, monitoring

    progress and

    determining what

    needs to be changed




    solutions &



    the Project:

    Evaluating success

    and documenting

    what was learned

    Planning the


    Determining the

    steps to solve

    the problem



    • Resolve operating problems

    • Conduct after action review

    • Reward & reassign team members

    • Define goals & objectives

    • Identify initial resources

    • Commit organization to goals

    • Develop work breakdown structure

    • Create project plan

    • Define targets

    • Maintain project status

    • Identify problems

    • Adjust targets & re-plan

    * Planning occurs during this initial phase as well as through the rest of the project. As a manager it is essential to constantly plan and adjust the plan.

    Audit principles

    Planning is Important!

    • Planning answers questions such as:

    • What must be done?

    • Who will do it?

    • How will they do it?

    • How long will it take?

    • How much will it cost?

    Our goal is to minimize risk

    in our audit programs!

    Audit principles

    Execute and Manage the Plan

    • There are five main tasks during the execution phase. These include:

      • Maintain status

      • Conduct meetings

      • Adjust targets & re-plan

      • Manage resources

      • Communicate, communicate, communicate

    Follow the work breakdown structure

    Follow the Work Breakdown Structure

    • A work breakdown structure is the logical sequence of work tasks and activities that are designed to deliver results.

    • In developing the work breakdown structure, five questions should be answered:

    • What tasks must be done?

    • Who will do each task?

    • How long will it take to do each task?

    • What materials/supplies are required?

    • How much will each task cost?

    Audit principles

    Identify Risks and Plan to Respond

    • Risks include anything that can stand in the way of meeting our goals and objectives.

    • Keep aware that real world events conspire to change our plans constantly. Stay alert to these changes and continuously work with your team leader revise your plans and communicate these changes to your audit team.

    W. Edwards Deming has pointed out that, even when the fire department puts out a fire, you are no better off after the fire than you were before it. Fire prevention is far better than fire fighting.

    Audit principles

    Recognize Team Formation

    A newly formed team needs structure to help them through four natural stages. Understand these stages are necessary, natural, and will eventually lead to great teamwork!

    • Performing

    • High morale and team identity forged by conceptual agreement and working patterns

    • Leader becomes team member

    • Forming

    • Conversations tend to stay at surface level

    • Heavy reliance on team leader

    • Storming

    • Conflict surfaces, usually around goals, roles and objectives

    • Need for strong problem solving skills

    • Norming

    • Members begin to agree on objectives and share feelings

    • Leader becomes more of a mentor and coach

    Audit principles

    Where do I Fit?

    Outside influences directly affect your team. It is necessary to work through these to create a successful on-time audit that satisfies goals and objectives and is within budget.

    Core Audit Team



    • A

    • Successful Audit

      Completed On Time

      Satisfies Goals and Objectives

      Within Budget

      Performed within CMEP rules



    Decision Makers




    Organizational Dynamics

    Budgetary Constraints

    Audit Team

    Resource Constraints

    Time Constraints

    Key Stakeholders

    Audit principles

    General Process

    • A kick-off meeting should be held with each entity

      • Introduce project team and entity contacts

      • Discuss audit plan and requirements

      • Discuss information needed and communication protocols

    • Regular status meetings should be held with audit team and the registered entity

    • Status reporting is important to help keep to the schedule and the budget

    • Maintain interview and information request processes

    • Identify and resolve issues and conflicts and use escalation process when necessary

    • Focus on the plan

    Audit principles

    Tools – Status Reporting

    Status Reports should be prepared on a regular basis in order to manage and maintain relevant information such as:

    • Accomplishments this Period

    • Activities Planned but not Accomplished

    • Activities Planned for next Period

    • Issues for Management Attention

    • Proposed Resolution to Issues

    • Action Items

    • Key Stakeholder Involvement

    • Important Communication Activities

    Status reporting template

    Status Reporting Template

    Audit principles

    Project Management

    • 1. Several project team members do not seem to get along with each other. As audit lead you should:

      ___ Bring it to their attention immediately that it is negatively impacting the team and try to find out more reasons why

      ___ Replace the less senior team member causing the issues

      ___ Tell them to go have a discussion and iron out their differences – “knock it off” and get along

      ___ Let things play out as long as the project runs smoothly and bring it up when the project is done in their evaluation

    Audit principles

    Project Management

    • As long as you have a detailed work plan regular project meetings are ___ necessary ___ not necessary to keep the project on time and on-track.

    • 3. The audit lead should ___ have ___ not have interim progress updates on the project with the registrant.

    Leveraging project management tools and techniques for audit success

    Leveraging Project Management Tools and Techniques for Audit Success

    • Questions

    • Section review

    • Follow-up items

    Module 9

    Module 9

    Basic Report and Work Paper Review

    Learning objectives7

    Learning Objectives

    • What do the standards require for work paper and report review?

    • What are tools that can make this process robust?

    • How do we link work papers to reporting for ease in future use in potential regulatory hearings?

    • What should the process be for work paper completion and review?

    • What should the process be for report preparation and review?

    Report and work paper review

    Report and Work Paper Review

    • Project Close - File Review

      • File should bear evidence of review by supervisory staff

      • Review checklists are a necessary tool to ensure not only documentation requirements but to make sure nothing is missed

      • Review procedures should include make sure:

        • All replies to representation letters have been received and are properly dated and signed by the appropriate level of management

        • All checklists have been signed off and completed

        • Proper sign-offs have been made at the proper level

        • Staff evaluations and feedback have taken place

        • All pertinent points are moved to the file for the next audit

    Report and work paper review1

    Report and Work Paper Review

    Report and work paper review2

    Report and Work Paper Review

    • File Review – What should we take out of the file?

      • “To-do” lists should be removed from the file prior to issuance

      • Review comments should be removed from the file

      • Superseded memos or information should not be included in the file

      • Always ask yourself, “This file is subject to legal discovery, what information is relevant to the report that we issued based on the evidence we collected and examined?”

    Report and work paper review3

    Report and Work Paper Review

    • Quality Assurance – File Control

      • Lock out file access after a time certain period-prevent the potential for manipulation of findings

      • Retention periods – follow organizational directives and CMEP rules – should be the longer of five years or audit cycle for retention

    Report and work paper review4

    Report and Work Paper Review

    • Work Paper Linkage

    • There must be linkage between every report finding back to an evidential work paper in order for it to be a credible and defensible finding

    • Think “if I were on the stand how would I document and defend this finding?”

    • Leave a trail you can find in the future to take you back to your thinking at this point in time….leave some breadcrumbs

    Report and work paper review5

    Report and Work Paper Review

    Work Paper Linkage

    Report and work paper review6

    Report and Work Paper Review

    Work Paper Linkage

    Report and work paper review7

    Report and Work Paper Review

    Work Paper Linkage

    Report and work paper review8

    Report and Work Paper Review

    Work Paper Linkage

    Report and work paper review and quality control

    Report and Work Paper Review and Quality Control

    File Review, Report Preparation and Issuance Process

    Report and work paper review9

    Report and Work Paper Review

    File Review, Report Preparation and Issuance Process

    Report and work paper review10

    Report and Work Paper Review

    • File Review and Reporting

      • Audit standard is that a professional not involved with the audit must be able to reach the same conclusion as the auditor – using the same information

        • Does the file you’re reviewing meet this standard?

      • File evidence must meet the requirements of “Audit Documentation”

      • Report should not be issued in final until the file has been reviewed and all sign-offs and quality assurance procedures have taken place

      • Hard to teach “quality assurance” for report review – get your most thorough and inquisitive reviewer on it for the best results

      • Utilize an internal inspection process to keep file issuance robust and meeting standards

    Report and work paper review11

    Report and Work Paper Review

    • Quality assurance reviews of your report are not necessary because only your audit team members can interpret the audit evidence to come to the conclusions in the report.

    • True False

    • 2. In a regulatory hearing regarding your report findings your recollection of interviews

    • will

    • will not suffice as supportable evidence.

    Report and work paper review12

    Report and Work Paper Review

    • 3.Once we have reviewed a compliance area and found no issues the next audit should

    • Exclude

    • Include the same area for review.

    • 4. Audit files should

    • Never be shredded

    • Shredded in accordance with organizational standards.

    Report and work paper review13

    Report and Work Paper Review

    • Questions?

    • Follow-up items

    Module 10

    Module 10

    Conflict Resolution and Escalation Protocols

    Learning objectives8

    Learning Objectives

    • Conflicts are natural and expected

    • How to handle issues and conflicts

    • How and when to escalate due to issues or lack of performance

    • Focus on resolution and moving forward

    Successful audits

    Successful Audits…

    • Successful management of audits always requires informed, proactive and timely management of issues.

    • Conflicts and Issues:

      • Are resolvable with action items

      • Can be escalated

      • Are proactively discovered during the course of the audit

      • Your leadership team will need to analyze a myriad of concerns and issues

        • Audit scope

        • Shortage of resource issues

        • Differences in interpretation, intent and styles

    • Good issue management and escalation processes should result in timely resolution of issues and conflicts!

    Different conflicts treat differently

    Different Conflicts Treat Differently

    • Conflicts related to the conduct of audits can be escalated for discussions through an informal process using senior management.

      Conflicts or disagreements with findings (potential violations) or not providing necessary information are not subject to the same escalation procedures (the CMEP has escalation and due process procedures which the Regional Entity staff person should reference).

    Issue management

    Issue Management

    • The issue management process involves monitoring the status of each of the concern, issue or conflict. The issue management process is depicted as follows:

    Clarification &













    Check for





    Involve all




    Specify scope


    Document progress

    Assign owner

    Find solution

    Set due date

    Plan implementation

    Issue management1

    Issue Management

    • All issues should be logged and reviewed with the Audit team.

      Most issues will be able to be resolved within the team. However, if the issues are beyond the control of the team, or impede the progress of the audit, it may be necessary to use anescalationprocess.

    Escalation is sometimes needed

    Escalation is Sometimes Needed

    • Sometimes issues will need to be escalated to get resolution

    • Reasons for escalation are:

      • If the issue is ‘Mission Critical’

      • Issues beyond the authority of an individual or team and require a consensus decision

      • Owner of the issue is not clear or cannot be established

      • Issues which are not being properly, addressed and may impede the progress of the audit (will affect the audit scope, costs and/or timeline)

    • Your leadership team will evaluate the issue and determine the appropriate steps to ensure that the issue is resolved.

    Why escalate

    Why Escalate?

    • Potential events requiring escalation

    • Missed deadline by 24 hours

    • Lack of meeting or interview attendance with no notice

    • Lack of access to necessary IT systems

    • Interview or meeting does not occur within 5 business days of original agreed to date

    • Refusal to supply information

    • Incomplete information

    • Lack of returning calls after initial call and follow up call

    • Remind the registrant that you must make a determination of compliance and failure to produce sufficient evidence to make such a determination will result in an alleged violation

    What is required

    What is Required?

    • All information and meeting requests (including interviews) will be documented in a log by requesting individual.

      Each corresponding escalation will be documented in a log which will be made available to project management and the registrant on a regular basis. The documentation should include:

      • Requested information or meeting

      • Requestor

      • Registrant individual contacted

      • Response date or date information received

      • Follow-up dates

      • Escalation protocol steps

    Escalation log

    Escalation Log

    Don t forget

    Don’t Forget

    • Keep to the facts

    • Assume the best

    • Involve your team to reach the best solution

    • Conflicts and issues are a natural part of the process – don’t look away!

    • Timely resolution of issues is needed so that they don’t fester and create new issues

    • Registrants can always use the due process protections of the CMEP

    • Once a decision is made or a conflict resolved, move forward!

    Conflict resolution

    Conflict Resolution

    • The registrant’s Compliance Manager comes to the audit lead and says an audit team member is “too nosy and pushy” in performing the audit. The audit lead should:

      ___ replace the team member

      ___congratulate them on a job well done

      ___ coach them to make sure their approach is collaborative yet gets the audit evidence needed

    Conflict resolution1

    Conflict Resolution

    • 2. The registrant’s Compliance Manager is a “good guy” yet constantly misses deadlines in getting information to the team. The audit team members should

    • ___ cut him some slack in meeting deadlines while encouraging him to get them the information

    • ___document these issues and follow the escalation protocol

    • Any instances involving continuous use of the escalation protocols

    • ___ should

    • ___ should not be part of report findings

    Conflict resolution and escalation protocols

    Conflict Resolution and Escalation Protocols

    • Questions

    • Section review

    • Follow-up items

    Module 11

    Module 11

    Audit Close Activities, Lessons Learned and On-going Performance Management

    Learning objectives9

    Learning Objectives

    • Why is closeout important?

    • What is an exit meeting?

    • Why collect and discuss Lessons Learned?

    • Is performance management an annual process?

    • What are some possible audit performance measures?

    Why a closeout phase

    Why a Closeout Phase?

    • It is important to ensure that a project is properly closed for two reasons:

      • First, there is a tendency for projects to drift on and on, and never end.

      • Secondly, it is important to ensure that the work of the audit team is acknowledged and that the lessons to be learned from the project are formally investigated and recorded for use on the next project.

    • Lack of closure leaves everyone feeling dissatisfied and unrewarded for the work (often extra work) that they have done.

    The closing phase

    The Closing Phase

    • The purpose of the Closing Phase is to formalize the acceptance of the final audit results, share knowledge gained during the project and bring the project to an orderly end.

      Key Steps include:

      • File and/or return to the registered entity all working or final documents

      • Complete internal final Quality Assurance Review

      • Have the team complete their self-evaluation forms

      • Hold Close-Out Meetings with the registered entity and with the project team. Lessons Learned from each meeting should be documented and shared with the project team.

      • Ensure all outstanding issues or tasks that were identified prior to and during the Close-Out Meetings have been resolved or a course of resolution has been determined.

      • Complete the performance review process for all audit team members

      • Lockout the audit file

      • Organize and hold a closing celebration for the audit team to celebrate the success of the project

    Exit meeting with registrant

    Exit Meeting with Registrant

    • Your closing presentation at the exit meeting should include:

    • Preliminary audit findings, with strong emphasis on the factual basis for each compliance or noncompliance item identified by the audit team, review CMEP due process protections and next steps for any findings (alleged violations),

    • Solicit feedback on how the audit was performed and any closing comments from the entity. Be sure to give the audited entity an opportunity to voice any challenges to facts as understood by the team. This should be reinforced when the audited entity has a draft report to review. Also, provide the audited entity with the opportunity to review the conclusions and supporting factual basis.

    • If it is your organization’s policy - present informal recommendations for improvements verbally related to your observations during the audit

    Lessons learned for the audit team internal

    Lessons Learned for the Audit Team (Internal)

    • When the audit is complete, the team should hold a review meeting to discuss the audit and identify…

    • What went well

    • What did not go well

    • What to replicate in future audits

    • What to change

    • Without assigning blame, the members conduct the audit review with the goals of learning from mistakes and improving future projects. Often called a postmortem, the audit-review meeting sometimes takes place shortly before the end of the project rather than afterward, because team members often must leave the project shortly before it ends.

    • The team documents suggestions for change as action items in the next audit plan.

    Lessons learned continued

    Lessons Learned (continued)

    • Allow the team time to reflect on and prepare for the review.

      • Ensure there is plenty of time for group discussion.

      • Keep the meeting positive

      • “Doing better next time” is the theme of the review

      • Document and share the results of the meeting



    • Be sure to celebrate in some way the end of the audit – we all want to be appreciated!

    Electric Reliability Organization

    Regional Entity Management Group

    On going audit performance management cycle

    On-going Audit Performance Management Cycle

    Review Audit Plans

    “What do we want to achieve?”

    “Set SMART goals – specific, measurable, achievable, realistic and timely goals

    Set Performance Goals


    Meet and Discuss expectations

    “What are growth areas and needed skills?”

    MonitorandEvaluate Performance

    “How much progress are we making toward our goals?”

    Is this just an annual process

    Is This Just an Annual Process?

    • NO!

    • It is important to routinely throughout the year discuss performance. It is also important to discuss performance on each (or larger) audits.

    • Be sure to….

      • Establish performance goals and expectations for a specified period of time during an audit.

      • Provide feedback regarding the performance to the stated goals and expectations as well as to the standards of good, solid performance at a particular level of responsibility.

      • Establish specific actions required to improve or demonstrate performance in specific areas.

      • Reinforce employees strengths that should be recognized and maintained.

    Possible measures

    Possible Measures

    • Quantitative measures might include:

    • Plan completion:This is a measure at a point in time. This may be measured using the number of projects completed, weighted by the planned size of each project, with estimates for projects in-progress. Measured throughout the year, it is compared against the percentage of the year elapsed.

    • Report issuance:This is a measure of the time elapsed from completion of testing to issuance of the final audit report, including management’s action plans. This can be measured in average days or percentage of reports issued within a certain standard, such as 30 days.

    • Issue closure:The number of days that reported issues remain open, or open after their agreed-upon closure date, are key measures.

    • Staff utilization rate:This is measured as the percentage of time spent on audit, as opposed to administrative time such as training or vacation.

    Retain your team

    Retain Your Team

    Developing and retaining quality professionals is a key concern!

    • Key methods for developing and retaining internal audit staff personnel include:

      • Provide challenging, varied assignments

      • Ensure great supervision

      • Have staff participate in audits from start to finish, to learn all phases of the audit process

      • Provide opportunities to lead (in-charge) audits, starting with smaller, easier audits

      • Involve staff in improvement task forces, such as preparation for quality assurance review

      • Have them participate in the recruiting and interviewing process for new hires

      • Rotating through various audit teams

      • Provide both outside training (e.g., seminars) and in-house training for two weeks/year

    Audit close process

    Audit Close Process

    • The audit file should always be open for audit findings. One never knows when instances of non-compliance will be communicated

      True False

    • An audit debrief meeting should be held just between the audit lead and organization upper management

    • True False

    Audit close process1

    Audit Close Process

    • Goals should not be established up front before the audit for audit team staff.

    • True False

    • 4. Audit project lead time is better spent interacting with the registrant vs. assisting staff in executing the work plan.

    • True False

    • 5. Audit staff should be capable of reading and understanding the audit work plan and executing the audit steps without a great deal of supervision or questions.

    • True False

    Regional Entity Management Group

    Audit close process2

    Audit Close Process

    • Questions

    • Section review

    • Follow-up items

  • Login