audit principles
Skip this Video
Download Presentation
Audit Principles

Loading in 2 Seconds...

play fullscreen
1 / 232

Audit Principles - PowerPoint PPT Presentation

  • Uploaded on

Audit Principles. NERC Auditor Training Introduction to Audit Principles and Techniques. REMG Compliance Auditor Training. Please Remember…. This is not technical training. There are “hard” and “soft” skills to auditing

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Audit Principles' - sutton

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
audit principles
Audit Principles

NERC Auditor TrainingIntroduction to Audit Principles and Techniques

please remember
Please Remember…
  • This is not technical training. There are “hard” and “soft” skills to auditing
  • This course is focused on auditing skill building and is based on years of auditing knowledge.
  • ALL auditing follow the same basic principles. We ALL need to think like auditors.
  • Discussion is welcomed – especially on how to apply standard practices to NERC compliance audits
  • Training materials are under development, so your feedback is very much appreciated!
  • Trainers assume that you have read the CMEP
  • 10.00 Introductions/Course Overview All
  • 10.30 Module 1 - Conceptual Framework Russ Hissom
  • Dan Skaar
  • 11.15 Module 2 – Audit Process and
  • Planning Russ Hissom
  • 12.00 Lunch
  • 1:00 Module 3 – Managing Relations Carol Arneson
  • 1:30 Module 4 – Audit Documentation and
  • Evidence (and RSAW Breakout) Russ Hissom
  • Kevin Goolsby
  • 2:30 Module 5 – Workpaper Preparation and
  • Information Requests Carol Arneson
  • 3:00 Break
  • 3.45 Module 6 – Audit Testing and Testing Methodologies Russ Hissom
  • 4.45 Questions and Discussion
  • 5.00 Happy Hour!!
  • 8.00 Module 7 – Interview Techniques Carol Arneson
  • 9:00 Module 8 – Leveraging Project Management
  • Tools and Techniques for Audit Success Carol Arneson
  • 9:30 Break
  • 9:50 Module 9 – Report and Workpaper Review Russ Hissom
  • 10:20 Module 10 – Conflict Resolution and Escalation Protocols Carol Arneson
  • 10.45 Module 11– Audit Close Activities, Lessons
  • Learned and On-going Performance
  • Management Russ Hissom
  • Close Questions and Discussion – Course Evaluations
compliance program assistance leadership team
Compliance Program Assistance Leadership Team

Virchow Krause Team Members

Carol Arneson, PMP, MBA, Senior Manager in the Energy and Utilities Group, has worked in the utility industry since 1976 specializing in financial and operational needs. She has broad utility experience at two Fortune 500 utility companies where she managed financial, cost management, strategy and business planning, generation support processes, and various other business processes for over 20 years. Carol has managed numerous projects serving municipal and investor-owned utilities including contract compliance audits, energy management services contracting and performance audits.

Russell Hissom, CPA, Partnerin the Energy and Utilities Group, specializes in serving the financial and operational needs of the utility industry. He has extensive experience with financial audits of utilities, management audits for utilities and State Public Utility Commissions, developing utility cost of service and rate design studies, analyzing the input and performance of parties under jointly owned electric generation contracts, assisting with accounting issues under FAS 133/149 and performing operational reviews.

don t underestimate the power that lack of compliance of rules and regulations has on any industry
Don’t underestimate the power that lack of compliance of rules and regulations has on any industry
  • History has shown us that compliance (or lack of compliance) has the power to do many things:
    • Financial collapse large listed companies
    • Reduce market capitalization by billions of dollars for alleged, egregious violations
    • Bankrupt companies
    • Closure of plants
do you remember
Do You Remember……..
  • Enron (SEC compliance)
  • Worldcom (SEC compliance)
  • Ameranth (SEC compliance)
  • Bear Sterns, Lehman, AIG, Wachovia, Merrill Lynch, etc. etc. (SEC compliance)
  • Southwest Airlines (FAA compliance)
  • Northeast blackout of 2003 (NERC voluntary compliance)
  • Southern Florida blackout of 2008 (NERC mandatory compliance)
crisis leads to regulation
Crisis Leads to Regulation
  • Blackouts
    • Energy Policy Act of 2005 (section 215 of the Federal Power Act) mandated standards in USA with financial penalties (separate agreements within Canadian jurisdictions)
  • Financial Collapse and Fraud
    • COSO (internal controls)
    • Sarbanes-Oxley (internal controls, governance, civil penalties on corporate officers)
role of the auditor
Role of the Auditor
  • Auditor’s responsibilities are much more important today than the past given the impacts from non compliance
  • Can you imagine if you, as an auditor, missed a major finding and had you made the finding you could have prevented an incident on the bulk power system? What about Enron, what if the auditor, early in the scandal discovered and reported the irregularities, perhaps things would be very different!
  • ALL auditors’ competency and training must be at a very high level to assure their duties to the industry and maintain the public trust.
  • Auditors’ work must stand up to public scrutiny and legal challenges!
module 1
Module 1

Conceptual Framework for Auditors

learning objectives
Learning Objectives
  • Understand what an “audit” is and is not
  • Develop the confidence to perform a competent audit
  • Understand the basic steps in an audit
  • Who are the audit standard setting bodies
  • What kinds of audits are there?
  • What does the technical guidance tell us to do?
what is an audit
What is an Audit?
  • An audit is an evaluation of a person, organization, system, process, project or product
  • It is not an investigation

Audits may not presume a potential violation; investigation presume a potential violation exists. Skills are similar in the conduct of an investigation.

Audits are performed to ascertain the validity and reliability of information, and may include an assessment of a system\'s internal compliance environment. The goal of an audit is to express an opinion whether some one or some entity meets a “standard” or does not meet a “standard” based upon a systematic review and testing of records. Due to practical constraints, an audit seeks to provide only reasonable assurance that the registrant is compliant with the applicable Reliability Standards.

necessary skills
Necessary Skills
  • Attention to detail
  • Good understanding of audit risks
  • Ability to work with people and experts
  • Subject matter expertise
  • Deep knowledge of reliability standards applicable to entities being audited
  • Knowledge of government auditing standards that apply to performance audits
  • Task management skills
  • Clear and concise communications
  • Ability to follow a standardized program
  • Good planning skills
  • Team player
  • Willingness to identify issues and be proactive in bringing them to attention
major steps involved
Major Steps Involved
  • A typical performance audit project involves the following steps:
    • Establish and communicate the scope and objectives for the audit.
    • Develop an understanding of the organization under review. This includes objectives of the audit, measurements, and key requirements. Review pertinent documents and interviews.
    • Identify control procedures used to ensure each key activity type is properly controlled, monitored and documented. Upfront, an internal compliance survey should be completed by the Registered Entity.
    • Develop and execute a risk-based sampling and testing approach to determine whether the most important activities are operating as intended.
    • Report findings and areas in compliance.
    • Complete audit closing tasks, review staff and start to prepare for your next audit.
tips for success
Tips for Success
  • Be the “ultimate” professional
  • Expect to be “monitored”– lead by example
  • Use empathy – remember what it’s like to sit on the other side of the table. Be compassionate, but firm
  • Remember you’re there to complete a job – not solve the auditee’s problems
  • Be proactive
  • Know the project work planbeforeyou begin
  • Communicate with your audit leader
  • Thoroughly document all testing and findings with quality evidence
audit types
Audit Types
  • Financial Audit
    • A financial audit is an independent assessment of the fairness by which a company\'s financial statements are presented by its management
      • Authoritative standard bodies include GAO, Canadian versions as well
  • Compliance Audit
    • A compliance audit is an independent assessment of the compliance by an entity with various laws or regulatory requirements
      • Authoritative bodies include GAO (e.g. chapter 7)
      • NERC compliance audits
  • Management Audit
    • A management audit is an independent assessment of the efficiency in various operating areas by an entity
      • SAS 70 audit, “agreed upon procedures”
value of audits
Value of Audits
  • Audits are not just checking if things happened or if compliance requirements were met – they can be used to provide great value to the registrant
    • Informal recommendations for process improvements or how to meet compliance requirements are a natural by-product of an audit (orally during the exit interview)
    • Audits serve the public and industry interests; there is a reliance on auditor’s work to identify compliance and non compliance; it’s the responsibility of the entity to comply and take necessary action to be compliance with standards
auditing concepts techniques
Auditing Concepts & Techniques
  • What is GAGAS?
  • In the United States – this is the standard for government performance audits – Generally Accepted Government Auditing Standards – GAGAS – aka the “yellow book”
  • Standards maintained by the Government Accountability Office (GAO)
  • GAGAS standards incorporate other standard bodies work
  • Requires auditors to serve the public interest and honor the public trust
  • Auditors must perform all duties with integrity, be independent and honest and candid with the entity being audited
  • Auditors should always exercise professional judgment and skepticism
gao chapter 2 ethical principles
GAO Chapter 2 Ethical Principles
  • Ethical principles guide the work of auditors
      • The public interest
      • Integrity
      • Objectivity
      • Proper use of information
      • Professional behavior
  • Please take five minutes and read Chapter 2.

Any thoughts?

gao chapter 3 general standards
GAO Chapter 3-General Standards
  • Independence
    • Free of conflicts
    • Appearance and in fact
  • Professional Judgment
    • Knowledge, skills, experiences,
    • Reasonable care
    • Professional Skepticism
    • Due diligence
  • Competency
    • Blend of education and experience
    • MUST have skills to perform audit
auditing concepts techniques1
Auditing Concepts & Techniques
  • Review GAO Chapter 3
  • What are the key applicable parts of Chapter 3 in the conduct of our audits?
professional skepticism
Professional Skepticism
  • The ability to approach any situation with a skeptical view towards conclusions reached without examining all factual data and using that data to verify and support your conclusion as an audit
  • me!
due diligence
Due Diligence

Performance Improvement Intent

Production & Reserve Increases

Per Person

Market Share

Retail Return



Increase %

Production Increase

per Person






Net Income $mm

Reserves/10 per




per Person











Net Income

Market Share Change

Strategic Targets/Initiatives

% of Capital



Quality Rating










Change in

Market Penetration

Capital Expenditures

Natural Gas




Increase in Gross

National Product

Refining Gross Operating






















The act of researching all available data to support a conclusion or position about an activity or outcome

due professional care
Due Professional Care
  • “Due Professional Care in the Performance of Work”
    • What is Reasonable Assurance?
    • Auditor must plan and perform audit to obtain appropriate evidence so that audit risk is limited to a low level appropriate for expressing an opinion on the assertion tested (making a compliance determination)
    • Absolute assurance may be not possible because of the nature of audit evidence. Point of distinction between an audit and investigation: an investigation may require absolute assurance for prosecution of a violation; therefore, “stacking of evidence” on a potential violation may be appropriate during an audit after a potential violation is discovered .
    • Materiality: is it material? Not all things are the same!
  • Management of the registrant is responsible for assuring compliance to Reliability Standards
auditing concepts techniques2
Auditing Concepts & Techniques
  • GAGAS Continuing Education Requirements (per Government Auditing Standards)
    • Applies to external and internal auditors who perform GAGAS audits
    • Standards require 80 hours every 2 years of continuing education – 24 hours in subjects directly related to the governmental environment or governmental auditing
      • Remaining hours should be in topics that directly enhance the auditor’s professional proficiency to perform audits
      • At least 20 of the 24 hour requirement should be done in a single calendar year
    • Auditors who do not supervise audits or who charge less than 20% of the annual time to audits need 24 hours every 2 years
    • Assume these requirements apply to NERC—stay sharp, be a life long learner in this profession!
auditing concepts techniques3
Auditing Concepts & Techniques
  • Review GAO Chapter 7-Field Work Standards for Performance Audits
  • What are the key applicable parts of Chapter 7 in the conduct of our audits?
      • Audit Evidence
      • Audit Risk
      • Audit Planning
      • Internal compliance environment
      • Sufficiency of audit evidence
why do auditor s fail
Why do Auditor’s Fail?
  • SEC and others have reviewed audits and auditors and have determined several reasons why auditors fail:
    • Failure to obtain sufficient evidence to support conclusions.
    • Failure to maintain independence
    • Failure to follow-up on unusual events (exercising professional skepticism)

Determine if the audit team is independent

  • “In all matters relating to the audit work, the audit organization and the individual auditor, whether government or public, should be free both in fact and appearance from personal, external, and organizational impairments to independence”
  • If it feels like you’re not independent, you’re not
  • If someone asks you “are you sure you’re independent?” - You’re probably not!
other independence matters
Other Independence Matters
  • Free of conflicts
  • NERC conflict of interest policy
  • Impairment of independence
    • Receiving gifts
    • Favors
    • If you think it is an impairment, it probably is an impairment
  • Must be independent “in appearance” and “in fact”
auditing concepts techniques audit evidence1
Auditing Concepts & TechniquesAudit Evidence
  • Audit Procedures for Obtaining Audit Evidence
    • Inspection of records or documents
    • Inspection of tangible assets
    • Inquiry
    • Confirmation
    • Recalculation
    • Re-performance
    • Analytical procedures
auditing concepts techniques4
Auditing Concepts & Techniques

Audit Risk and Materiality in Conducting an Audit

Inherent Risk (IR) – the risk linked to the activity itself assuming there are no related controls

  • Example: Registrant performs activities linked to the bulk power system operations and planning. Registrant has no documentation and not trained staff to perform requirements under a Reliability Standard
  • Others?
  • ¹A material misstatement under Reliability Standards would be a requirement under a standard is not being met and it’s a material impact to the bulk power system or has the potential to materially impact the reliability of the bulk power system.
auditing concepts techniques5
Auditing Concepts & Techniques

Audit Risk and Materiality in Conducting an Audit

Control Risk (CR) - the risk that controls will not prevent, detect and correct errors

  • Example: Registrant has documentation and trained staff, but no evidence of adequate supervision or review.
  • Others?
auditing concepts techniques6
Auditing Concepts & Techniques

Audit Risk and Materiality in Conducting an Audit

Detection Risk (DR) – risk that auditor will not detect a material misstatement – function of audit procedure and its application by the auditor

  • Example: Regional Entity sampling of a requirement did not include enough samples. Result was a material number of samples which did not meet requirements (e.g.. non-compliant) were not detected due to insufficient sampling method or sample size.
  • Others?
auditing concepts techniques7
Auditing Concepts & Techniques

Understanding the Entity and Its Environment and Assessing the Risks of Material Misstatement (Compliance Environment)

  • Auditor should obtain an understanding of the entity’s internal compliance environment using the NERC internal compliance survey as the framework work; the framework includes:
    • Control environment (compliance)
    • Risk assessment
    • Information and communication systems
    • Control activities
    • Monitoring

Auditor should send an internal compliance survey out before the audit to ascertain the relative risk; this is normally done in other audits which use a risk based approach

elements of a good internal compliance program
Elements of A Good Internal Compliance Program
  • Requires self assessments and/or self auditing
  • Encourages self-reporting
  • Directs aggressive, timely corrective actions
  • Provides documentation quickly
  • Demonstrates knowledge of the requirements of each applicable Reliability Standard
  • Maintains organization chart for internal compliance including senior management roles
  • Demonstrates independence from operations – a direct chain of command to senior management/CEO
  • Establishes internal compliance training program
  • Disciplinary procedures for deviation from compliance
  • Controls to prevent reoccurrence of violation
  • Whistleblower policies
  • Strong transparency of process and event facts
  • A strong internal compliance program is an important mitigating factor in any enforcement action as well as helping assessing risk (refer to handout for example of survey)
basic auditing concepts
Basic Auditing Concepts
  • The three types of audit risk are:




  • Key principles of Chapter 2 of GAO:
basic auditing concepts1
Basic Auditing Concepts
  • Which is better audit evidence?
    • You read the policies and procedures manual of the registrant regarding their procedures in place to comply with PRC-005-01___?
    • The Compliance Manager of the registrant brags in an interview that the registrant’s procedures are so sound and controls so strong that “we can’t have an instance of non-compliance in this area and I’ll bet you lunch on it” ____?
auditing concepts techniques8
Auditing Concepts & Techniques
  • Questions?
  • Follow-up items
module 2
Module 2

Audit Process and Planning

preparing monitoring your audit
Preparing & Monitoring Your Audit
  • What is the “ideal” time allocation on an audit?

Audit Planning – 10% - 20%


        • Determine applicable Reliability Standards
        • Prepare budget and staffing
        • Information request development and issuance
        • Scheduling with Registrant
        • Travel arrangements
        • Determining materiality and areas of risk
        • Determining sample sizes
        • Follow-up on findings
        • Preliminary Registrant meetings
        • Review past history – past reports, alleged violations, enforcement actions, mitigation plans, etc.
audit planning and supervision
Audit Planning and Supervision
  • Planning the project – the Standard

Planning must be documented

      • Professional judgment should be used
      • Work plans should be established
      • Needs of potential users of the reports should be considered (regulators, the registrant, industry, public interests)
      • Auditor should understand what they are auditing
      • Controls around the area to be audited should be documented and understood
      • Procedures used should be specifically designed to test compliance and to detect non-compliance
      • Criteria needed to evaluate findings should be documented in the planning stage
audit planning and supervision1
Audit Planning and Supervision
  • Planning the project – the Standard

Planning must be documented

      • Previous audits and findings should be used to focus the work plan
      • Data needed should be identified and requested through an information request from its source
      • Use of other auditor work or specialists should be considered and impact of that evaluated
      • Staffing should be sufficient to get the job done
      • Management should be contacted about audit planning – work plan and audit strategy should NOT be discussed in great detail
      • Most communication with management should be in writing
      • Professional judgment is key
audit planning and supervision2
Audit Planning and Supervision
  • Planning the audit – the Standard

Planning should be documented

Staff should be assigned that:

      • Know the work they are doing
      • Know the subject matter
      • Have the appropriate communication skills
      • Meet the appropriate continuing education requirements

Staffing should be assigned

      • Staff should have the proper skills or “collective” knowledge base for the job
      • Assign enough staff to get the job done
      • Provide for on-the-job staff training
      • Bring in specialists when needed
audit planning and supervision3
Audit Planning and Supervision
  • Planning the audit

Pre-audit procedures

Organize data requests (e.g. via audit letter at least 60 days in advance)

Cross reference information to authoritative documents (Approved Reliability Standards)

Sufficient time to review documentation prior to field (site) work

audit planning and supervision4
Audit Planning and Supervision

This year’s experience becomes part of next year’s plan

  • Previous audits and findings should be used to shape the audit work plan and identify risks
  • Review pending mitigation plans
  • Must document that you properly planned the audit

Electric Reliability Organization

Regional Entity Management Group

prepare a project risk assessment
Prepare a Project Risk Assessment
  • Entity will not provide sufficient evidence
  • System event disrupts availability of subject matter experts
  • _____________________
  • _____________________
  • _____________________
  • _____________________
checklist of planning documentation minimum
Checklist of Planning Documentation (Minimum)

Review Reliability Standards and requirements applicable to the registrant

Review of how the registrant is registered (what functions?)

Any significant changes in operations (mergers, acquisitions, etc.)

Review of past compliance records, readiness evaluations, compliance reports, investigations and violations history

Mitigation plans

Surveys of other bordering registrants

Compliance audit letter

List of primary contacts

Logistics for the audit

Estimates of audit hours

List of external, outsourced contractors and their purpose

Documentation related to meeting audit team conflicts and independence requirements

Identification of audit team members and their roles on audit

Documentation that audit team reviewed current CMEP

Internal compliance survey

staffing the audit
Staffing the Audit
  • What is leverage and how can it help?

Leverage - Assigning audit activities based on level of experience

      • Goals:
        • Raise experience level of staff
        • Get project done in most efficient manner
        • Maximize resources, i.e. focus on high priority tasks for staff
        • Assign most efficient use of outsourcing resources
        • Meet annual audit scheduling needs
        • Help staff meet hours goals
staffing the audit1
Staffing the Audit
  • What is a good leverage model?

Experienced auditors

    • Research RE and get to know their business activities and relevant audit areas
    • Serve as RE contact
    • Audit planning and staffing determination
    • Design audit tests
    • Lead audit fieldwork and assign tasks
    • Conduct exit interview with RE
    • Review audit work papers – assign follow-up work to audit senior to clear items
    • Prepare high level sections of report
    • Review other areas of report
    • Be responsible for report content and findings
    • Prepare staff evaluations
    • Prepare expert witness testimony
    • Serve as expert witness
    • Train staff in leading smaller audits
staffing the audit2
Staffing the Audit
  • What is a good leverage model?

Senior auditors

    • Research registrant and get to know their business activities and relevant audit areas
    • Serve as registrant contact
    • Assist with audit planning and staffing determination
    • Assist in designing audit tests
    • Supervise audit staff in their fieldwork activities
    • Participate in exit interview with registrant – oversee follow-up questions
    • Complete audit work papers and review staff work papers – assign follow-up work to audit staff to clear items
    • Complete audit work papers and checklists
    • Prepare detailed sections of report and review other areas of report
    • Train and prepare staff evaluations
    • Assist in preparing expert witness testimony (if needed)
    • Gain experience in supervising and leading larger audits
staffing the audit3
Staffing the Audit
  • What is a good leverage model?

Staff auditors

    • Research registrant and get to know their business activities and relevant audit areas
    • Assist in audit planning
    • Performed detailed audit tests and sampling
    • Propose audit findings to audit leader
    • Attend exit interview with registrant and assist with questions
    • Complete audit work papers and checklists
    • Clear review comments on work papers as assigned
    • Assist in preparation of expert witness testimony (if needed)
    • Participate in evaluation meeting with audit leader or audit senior
    • Gain experience to become audit senior
audit planning
Audit Planning
  • Planning the audit

Use of a Specialist

    • You should be able to tell the specialist what needs to be done
    • You need to be able to identify whether what the specialist does meets your needs
    • You need to be able to evaluate the specialist’s results to apply to other areas of the audit
    • External specialists must be under the same independence and conflict rules of staff
audit planning1
Audit Planning
  • Some benefits:
  • A properly planned audit mitigates risk
    • Risk of failure includes losing certification to perform compliance audits, reputational risk
    • Proper planning will translate into fewer challenges in the findings
  • Proper planning demonstrates professionalism and competency
preparing monitoring your audit1
Preparing & Monitoring Your Audit
  • What is the “ideal” time allocation on an audit?

Audit Performance (Fieldwork, off and on site) – 60% - 70%


    • Pre-audit staff review of registrant information and evidence
    • Testing
    • Prepare detailed work papers to demonstrate findings
    • Discuss preliminary findings and information to clear them if needed
    • Closing conference/briefing with Registrant including Audit results
    • Identify open item list and responsibilities for delivery with Registered Entity
    • Write draft report
preparing monitoring your audit2
Preparing & Monitoring Your Audit
  • What is the “ideal” time allocation on an audit?

Audit Performance (Fieldwork) – 60% - 70% (focus on the value)

    • Design testing to gain adequate coverage for “high risk” areas
    • Leave the little stuff alone – use 80/20 rule
    • Define the goal of testing in each area and balance between findings and providing recommendations for solutions to registrant
    • Use fieldwork as the opportunity to train staff
    • Always get as much as possible done in the field – resolve open issues
    • If an alleged violation is discovered, auditor should determine: Is it a risk to bulk power system reliability?
audit performance
Audit Performance

Staff Supervision - Practice the W approach

  • Why are we doing this?
  • What are we supposed to get done?
  • Please show me how to do it….
audit performance1
Audit Performance
  • Staff are to be properly supervised
    • Ensure the audit objectives are accomplished
    • Provide guidance
    • Stay informed about problems encountered
    • Review the work
    • Train the staff
preparing monitoring your audit3
Preparing & Monitoring Your Audit
  • What is the “ideal” time allocation on an audit?

Audit Reporting – 30% or less


      • Clear open item lists
      • Complete work papers
      • Prepare draft report
      • Issue draft for internal review
      • Comply with internal and CMEP requirements
      • Issue draft report to registrant for their review and comment
      • Issue final report
      • Make notes for next audit of the registrant
      • Prepare staff evaluations
due professional care judgment
Due Professional Care & Judgment
  • Auditor must plan and perform audit to obtain appropriate evidence so that audit risk is limited to a low level appropriate for expressing a conclusion or position
audit process
Audit Process
  • Lunch provided on-premises by the registrant to the audit team is permissible. However, you can not go to a restaurant with the registrant and they pick up the tab
  • True False
  • 2. The best way to efficiently complete the audit is for the audit lead to do most of the hands-on testing and interviews
  • True False
audit process1
Audit Process
  • 3. Audit planning requires the audit lead to complete all checklists and information requests once at the registrant site and then wait for information to be provided:
  • True False
  • The registrant invites the audit team to join them for happy hour to discuss how the audit is going – does this impair independence of the team?
  • True False
audit process2
Audit Process
  • One of the registrant’s staffers just had a new addition to the family. One of the audit team members chips in for a gift. This:


does not impair independence.

  • The registrant’s staff seem like “good people”. One of the registrant says “don’t’ worry about testing PRC-005-01, we’re all over it”. The audit team is running short of time in the field and seeking for areas of minimal risk to pass testing on. PRC-005-01 is an area that

should be

should not be on the “to be rotated list” for registrant self-reporting.

audit process and planning
Audit Process and Planning
  • Questions
  • Section review
  • Follow-up items
module 3
Module 3

Managing Relations

Electric Reliability Organization

Regional Entity Management Group

learning objectives1
Learning Objectives
  • Learn keys to establishing good relations
  • Balance trust with audit responsibilities
  • Empathy and active listening
  • What to do when relationships are strained

Electric Reliability Organization

Regional Entity Management Group


What are the concerns of your registrant?1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

Electric Reliability Organization

Regional Entity Management Group

what are the concerns of your registrant
What are the concerns of your registrant?
  • Possible responses:
  • What are they really looking for?
  • How will this work?
  • How am I going to get all of this done and keep my ‘real job’ going?
  • Do they have the knowledge and experience to understand what we do?
  • If they find something, what will happen?
  • Will they keep my confidential information safe?
  • Perhaps if I go slowly, they will go away?
  • Will we receive a large fine?

Electric Reliability Organization

Regional Entity Management Group

how to establish good relations
How to establish good relations
  • Establish trust – use the Golden Rule
  • Suspend judgment
  • Communicate, communicate, communicate
  • Stick to the plan
  • Tell them what you are going to do, tell them what you are doing, tell them what you did
  • Use empathic responses
  • Practice active listening
  • Know your role
  • Communicate, communicate, communicate

Electric Reliability Organization

Regional Entity Management Group

how to establish trust
How to establish trust
  • Credibility builds trust
    • Knowledge
    • Skill
    • Competency
  • Share information
  • Be fair and honest
  • Meet your commitments
  • Listen well
  • Perform competently
  • Be predictable
  • Communicate openly and clearly

Trust is essential for conflict resolution – trust is a key component as it is associated with enhanced cooperation, information sharing and problem solving

Electric Reliability Organization

Regional Entity Management Group

use empathic responses
Use Empathic Responses
  • Empathy is often described as “putting oneself in another’s shoes” or…
  • “the ability to put oneself in the role of another”.
  • Empathy is not pity nor sympathy, but instead the ability to understand what it feels like to be the subject of an audit.

Electric Reliability Organization

Regional Entity Management Group

active listening
Active Listening
  • Active listening is an intent to listen for meaning.
  • The listener checks with the speaker to see that a statement has been correctly heard and understood. The goal of active listening is to improve mutual understanding
  • Watch body language
  • Paraphrase the speaker’s words
  • Let the speaker know that you are hearing what they are saying
  • Give the speaker your full attention – if you are doing other things, they will feel not heard.
  • Nod occasionally, smile and react
  • Encourage with small verbal comments (yes, oh, ummm…)
  • Make your posture open and inviting
  • Ask clarifying questions
  • Summarize what you have heard

Electric Reliability Organization

Regional Entity Management Group

what should you do if relations seem strained
What should you do if relations seem strained
  • Raise any concerns first with your team leader or manager
  • Do not assume that the root cause is your behavior – it may be other stresses in the auditee’s world – we all have a bad day
  • Proactively observe interactions and seek assistance
    • Involving other team members sometimes can help
    • Sometimes the “chemistry” between individuals just doesn’t work – don’t let it fester, seek help from your team leader
    • Be professional and objective; if there are genuine disagreements on “findings”; listen, consider, but, at the end of the day the auditor’s judgement and determination must be relied upon in the public domain; politely remind the registrant of due process protections through the enforcement processes (point to the CMEP)
    • If registrant feels auditors are acting unfairly, request registrant contact senior executive at Regional Entity; registrants do have a right to disagree; audit team members should not engage in “debates”; again, there are formal due process protections when there are “findings” (e.g. potential alleged violation)

Electric Reliability Organization

Regional Entity Management Group

managing relations
Managing Relations
  • You and the compliance manager of the registrant have not hit it off. You are scheduled to interview the compliance manager in connection with the field audit. You should
  • ___ Continue with the audit as planned
  • ___ Add another of the audit staff to the interview for support and note taking
  • ___ Cancel the interview and schedule someone else
  • ___ Ask another audit staff to do the interview

Electric Reliability Organization

Regional Entity Management Group

managing relations1
Managing Relations
  • 2. The registrant’s compliance manager has said to your boss that the report filed is a “bunch of bs and full of misstatements!” You should:
  • ____ Revise the report based on the compliance manager’s comments
  • ____ Add another finding to the report on the compliance manager’s lack of agreement with your report
  • ____ Point out the evidence found supporting your assertions to the compliance manager in the exit conference

Electric Reliability Organization

Regional Entity Management Group

managing relations2
Managing Relations
  • Questions
  • Section review
  • Follow-up items

Electric Reliability Organization

Regional Entity Management Group

module 4
Module 4

Audit Documentation & Evidence

learning objectives2
Learning Objectives
  • What do the audit standards require for documentation?
  • What’s the “yellow book”?
  • What do you need to do to meet GAGAS evidence requirements?
  • What types of audit work papers are there?
  • What is “quality” when it comes to audit tests?
audit documentation and evidence
Audit Documentation and Evidence
  • Auditors should prepare and maintain audit documentation. Audit documentation related to planning, conducting, and reporting on the audit should contain sufficient information to enable an experienced auditor, who has had no previous connection with the audit, to ascertain from the audit documentation the evidence that supports the auditors’ significant judgments and conclusions. Audit documentation should contain support for findings, conclusions, and recommendations before auditors issue their report.
audit documentation and evidence1
Audit Documentation and Evidence
  • Audit Documentation Standards
  • The form and content of audit documentation should be designed to meet the circumstances of the particular audit. The information contained in audit documentation constitutes the principal record of the work that the auditors have performed in accordance with standards and the conclusions that the auditors have reached. The quantity, type, and content of audit documentation are a matter of the auditors’ professional judgment.
audit documentation and evidence2
Audit Documentation and Evidence
  • Audit Documentation Standards
  • Audit documentation serves to:
    • Provide the principal support for the auditor’s report
    • Aid in conducting and supervising the audit
    • Allow for the review of audit quality
audit documentation and evidence3
Audit Documentation and Evidence
  • Audit Documentation Standards

Documentation should provide these items:

    • Objectives, scope and methodology of the audit, including sampling and other selection criteria
    • Auditors’ determination that certain standards do not apply or that an applicable standard was not followed, the reasons therefore, and the known effect that not following the applicable standard had, or could have had, on the audit
    • Work performed to support significant judgments and conclusions (including descriptions of procedures and records examined)
    • Evidence of supervisory reviews before the report is issued
audit documentation and evidence4
Audit Documentation and Evidence
  • Evidence and documentation in an audit must provide the following support to determine whether a registrant is compliant with a standard/requirement:
    • Existence or occurrence –recorded events or activities have occurred
    • Completeness –everything that happened is presented and events or activities presented are in accordance with industry standards
    • Accuracy and classification –events and activities presented are accurate and correctly presented
audit documentation and evidence5
Audit Documentation and Evidence
  • Basic GAO or “Yellow Book” procedures
    • Auditor mustidentifypertinent regulations (CMEP and NERC)
    • Auditor mustassessthe risks of materials non-compliance
        • Assess internal controls (compliance environment)
    • Auditor must design steps and procedures totestcompliance with regulations to ensure that both unintentional and intentional instances of material noncompliance are detected
    • Auditorissuesreport on tests of compliance in which all instances of noncompliance or violations must be reported
audit documentation and evidence6
Audit Documentation and Evidence
  • Audit Evidence
    • Any information that corroborates or refutes an assertion of compliance
  • Objective of Audit Documentation
    • Provide principal support for the representations in an auditor’s report
    • Assist in planning, performance and supervision of the engagement
    • Extraneous (un-needed) audit evidence not needed should not be included in work papers
audit documentation and evidence7
Audit Documentation and Evidence
  • “Sufficiency of Audit Documentation”
    • Enables members of the audit team with supervision and review responsibilities to understand the nature, timing and results of auditing procedures performed and evidence obtained
    • Indicates which audit team members did the work
    • Shows that the records examined agree with the assertions being tested
    • Acid test – if you have no connection with the audit, can you come to the same conclusions? Uninformed Reviewer Test
audit documentation and evidence8
Audit Documentation and Evidence
  • “Sufficiency of Audit Documentation”
    • All audit documentation should be complete before the report is issued
    • Any post-audit procedures should be dated and identified
    • Lockout report and findings 60 days after completion – no modifications after lockout; limiting access via lockout prevents potential manipulation of findings and reports
    • Audit document must support compliance or create the record for non-compliance; evidence must stand on its own merits
audit documentation and evidence9
Audit Documentation and Evidence
  • Retention of Audit Work Papers
    • General rules are five years
    • Follow reliability standards, if greater than five years or minimum audit cycle (whichever is longer)
    • Retain until controversy is resolved
  • Standards require adequately safeguarding audit documentation—make them secure!!
  • Standards require defined policies for release of audit documentation to outside parties in keeping with laws and regulations that apply to both the audited registrant and audit organization
audit documentation and evidence10
Audit Documentation and Evidence
  • Quality of Audit Evidence
  • Influenced by its Source and Nature
    • Knowledgeable independent sources
    • Generated internally when controls are effective
    • Directly obtained evidence by the auditor (observation) vs. inquiry
    • Documentation of events (i.e.. written logs vs. oral representation)
    • Original documents vs. reproduction (copies and fax)
    • SCADA/EMS screenshots
    • Emails or other electronic documentation
    • Operating procedures
    • Phone transcripts
audit documentation and evidence11
Audit Documentation and Evidence
  • Quality of Audit Evidence
  • Some evidence is better than others
    • Evidence obtained when internal compliance program and related controls are effective is better than when internal controls are not effective
    • Direct evidence (physical examination, observation, logs, records, computation and inspection) is better than indirect evidence (procedures)
    • Original documents are better than copies
    • Testimonial evidence is more reliable if it is given freely
    • Testimonial evidence from unbiased party is better
    • Evidence from credible third party may be more competent than that of management
audit documentation and evidence12
Audit Documentation and Evidence
  • Quality of Audit Evidence
  • Some evidence is better than others
    • Written representations from management confirm oral representations
      • Certifications from officers are required given the nature of the auditors work (similar to Sarbanes Oxley)
      • Representations can be upfront via a general certification or may be done during the audit on an as-needed basis
      • Can be used when other documentation is not available (can be in the form of an affidavit, if it’s a material matter)
    • Data can be gathered by auditors through their own observations and measurements (interviews, questionnaires, observation, computations)
audit documentation and evidence13
Audit Documentation and Evidence
  • Quality of Audit Evidence
  • Some evidence is better than others
    • Data can be gathered by management– auditor must determine if information is valid and reliable
    • Third parties can gather data – auditor must determine reliance on that data
    • Corroborate weaker evidence with more evidence to gain reasonable assurance
    • Should not base evidence quality or other decisions in the audit upon a potential monetary penalty. Suggest auditors separate a “finding” from the enforcement implications to keep it “clean”.

Regional Entity Management Group

audit documentation and evidence14
Audit Documentation and Evidence
  • Audit Procedures for Obtaining Audit Evidence
    • Inspection of records or documents
    • Inspection of tangible assets
    • Inquiry
    • Confirmation
    • Recalculation
    • Re-performance
    • Analytical procedures

** Breakout Exercise

need for quality in gathering and supporting audit evidence
Need for Quality in Gathering and Supporting Audit Evidence
  • Quality of Report is Supported only by the Evidence Gathered
gathering quality evidence
Gathering Quality Evidence

Know the RSAW

Know the standard you’re testing

What is the linkage?

Follow the RSAW testing requirements line by line

There MUST be evidence that supports every line item tested

What did you do? How did you do it? What piece of evidence did you select and why?


gathering quality evidence1
Gathering Quality Evidence

If you can’t explain why you selected a piece of evidence and how it supports your position, it isn’t good evidence – discard it (it will cause you problems later)

Every piece of evidence must have a direct link to the report or it should not be in the file

Conversely – there should be no unsupported statements in the report


writing a quality report
Writing a Quality Report

Reports should include:

A clear explanation of the requirements in terms that are understandable to the general public

A description of the method of testing compliance, including testing method used, population definition, sample size and results found

A record of interviews (date, time, participants, subjects covered) with direct quotes on pertinent topics

A description of audit results which demonstrate non-compliance with standards and meet standard of legal sufficiency

Explanation/argument which help the reader understand that the results (evidence) found equate to non-compliance

Documentation of results review with registrant

audit documentation and evidence15
Audit Documentation and Evidence
  • Main tool in auditing is “sampling”
    • Sampling is… a systematic and defensible approach to drawing a conclusion of a population based on reviewing less than 100% of that population
  • Why not sample 100% of the transactions?:
    • Lack of time and resources
    • How do you know you have 100% of the population?
    • Finding “one more” negates validity of the entire sample
    • Rely on more on systematic controls in place in some cases for reliance that compliance is being met
audit documentation and evidence16
Audit Documentation and Evidence
  • Sampling Methods




Population Proportional


audit documentation and evidence17
Audit Documentation and Evidence
  • What are the main tools used in compiling audit documentation?
    • Work programs (RSAWs)
    • Checklists
    • Digital files
    • Audit extraction tools
    • Manual work papers
    • Representation letters
    • Formal reports
audit documentation and evidence18
Audit Documentation and Evidence
  • Suggested File Construction (NERC and RE’s should establish a standard file construction)
    • Current files
      • Applicable standards and RSAW’s
      • Active mitigation plans
      • Pending violations
      • Internal compliance survey
      • Current self certifications
    • Permanent files
      • Documents overall registrant information needed for ongoing audits
      • Organization structure, corporate information
    • Archived files
      • Past audits and other materials
        • Findings communicated to management
        • Management responses
      • Self certifications
      • Completed mitigation plans
audit documentation and evidence19
Audit Documentation and Evidence
  • The yellow book is:
  • ____ A book where you can find good restaurants to visit after field work is completed for the day
  • ____ A guidebook from the GAO on audit standards
  • Audit Evidence is:

____ Any information that corroborates or refutes an assertion of compliance

____ Documentation that will be admissible in hearings for non-compliance

____ All of the above

audit documentation and evidence20
Audit Documentation and Evidence
  • 3. The objective of Audit Documentation is to:

____ Provide principal support for the representations in an auditor’s report

____ Assist in planning, performance and supervision of the engagement

____ All of the above

  • Lack of audit evidence will lead to:
  • ____ Embarrassment on the witness stand should you testify in hearings as to the assertions in your report
  • ____ Reversal of fines assessed for non-compliance by the RE
  • ____ All of the above
audit documentation and evidence21
Audit Documentation and Evidence
  • Lack of audit evidence in support of report assertions by your organization may lead to what types of actions?
  • _____ Reversal of fines assessed for non- compliance by the RE
  • _____ Revocation of the right to perform compliance audits by your organization
  • _____ All of the above
audit documentation and evidence22
Audit Documentation and Evidence
  • Questions?
  • Follow-up items

Compliance Monitoring and Enforcement Program Auditor Training

RSAW Workout


Compliance Audits

  • The following example is from an audit, the names and references have been changed.
  • Also following are tools and excerpts to help with the workout.

Audit Phone Trans-cripts


one transcripts for Midwest Electric Co.



002 R2, February 10, 2008




Step number 1


Uh, for the Fishie sub finished up 115kV bkr 5201 time 0806

Now you are ready for us to do this other part


Aren’t you?


Give me a call back

Are you in the clear?

I am in the clear





This is Josh

Line 3340 is out of service



Alright, thanks Jim


That would be clearance order 7901, give you a


Jesus yeah, 7



ME this is Josh

Hey Josh, this is Donny, we are going to start switching on 115 line number 7577a at


Alrighty, I will put her down



Audit Phone Trans-cripts


This is Josh


Yes, Josh uh, circuit 6742 at 40

and Lea is on non reclose and tagged for your





Control this is Josh

Hey Josh, this is Bob


Order 7086

Right on

Ok, step 5 install grounds at Treat Junction between both open points


Got all that

Got all that


audit example
Audit Example
  • Questions?
  • Section review and quiz
  • Follow-up items
module 5
Module 5

Work Paper Preparation and Information Requests

learning objectives3
Learning Objectives
  • What are the audit documentation standards?
  • What are the building blocks for work paper preparation?
  • How to prepare an effective information request
  • When to use third party confirmations
  • What is a “Compliance Audit or Get Ready” letter
  • How to track requests
work paper preparation
Work Paper Preparation
  • Audit Documentation Standards
  • Key characteristics
    • Sufficient and complete
    • Written
    • Support findings (compliance or non-compliance)
    • Professional judgment - quantity, type and content
  • Audit work papers are the principal record of the work performed by the auditor; work papers form the foundation for findings of compliance
work paper preparation1
Work Paper Preparation
  • Common Paperless Audit Work Paper Platforms
  • Word
  • Excel
  • Visio
  • Access
  • Adobe
  • TIF files
  • Data extraction tools
    • Does any Regional Entity staff have a data extraction tool? Are these necessary for a compliance audit?
work paper preparation2
Work Paper Preparation
  • Common Paperless Audit Work Paper Platforms
  • Audit standard setting bodies including GAO allow use of electronic work papers (in fact, electronic formats are essential to manage the volume of information)
  • Court cases allow submittal of electronic work papers into evidence
  • Regulatory bodies regularly use electronic work papers in hearings and proceedings
work paper preparation3
Work Paper Preparation
  • Use References, if necessary-Common References
  • CConfirmed
  • T Traced to Records (indicate type)
  • NA Not applicable
  • NN Not Considered Necessary
  • PY Ties to Prior Year\'s Working Papers
  • V Vouched
  • R Recalculated
  • CF Cross footed
  • F Footed
  • PI Physically Inspected
  • P Requested Positive Confirm
  • N Requested Negative Confirm
  • PF Ties to Permanent File
  • NW No Further Work Necessary
  • PV Possible Violation
confirmations and their use
Confirmations and their Use
  • Positive Confirmation
    • Used when confirming evidence purported by a registrant as being true
    • Requires action by the confirming party
    • Very reliable evidence
    • Follow-up needed for non-response by confirmee
    • Example-use to confirm that a proper action was performed by a BA from the RC; or vice versa
    • Positive confirmations are prepared by the auditor on their letterhead
    • Use judgment. Positive confirmation are useful to corroborate weak evidence
  • Negative Confirmation
    • Used when confirming evidence purported by the registrant as being true
    • Requires NO action by the confirming party
    • Less reliable evidence than positive confirmations
    • Follow-up needed for response that indicates disagreement by the confirming party with the evidence to be confirmed
work paper preparation4
Work Paper Preparation

Positive Confirmation (on auditor letterhead)

  • Date
  • Party to be confirmed
  • Dear xxxx:
  • Regional Entity, XXXX, are performing a compliance audit under section 215 of the Federal Power Act; as such Regional Entity is examining certain required communications and/or actions involved between Registrant and Confirmed Party. In connection with this examination, please confirm directly the correctness of the information shown below:
  • INFORMATION TO BE CONFIRMED (also reference standard and requirement)
  • A business reply envelope is enclosed for your convenience.
  • Sincerely,
  • XXXX Auditor
  • >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  • The information contained in this letter is correct to the best of our knowledge.
  • Signed – (Confirmed party representative)
work paper preparation5
Work Paper Preparation

Confirmation Tracking


Compliance Audit Letter

  • One of your first contacts with a registered entity is at least 60 days via a notice.
  • Information request is built off of your audit planning phase. It is intended to ask for pertinent data which will provide the total population from which you will draw your samples.
  • Often this request is included by a “Compliance Audit or Get Ready Letter”
  • NERC and Regional Entities have standardized the letter (refer to handouts)
compliance audit letter
Compliance Audit Letter
  • The key elements of Compliance Audit Letter are…
    • Includes certain representations and certifications (see examples-letter to Corporate Officer; letter to Primary Compliance Contact)
    • Advanced notice and reminder that it is time to get ready for the audit
    • Confirmation of scheduled audit dates
    • How you expect to receive the requested information (hard copies, electronic files, etc.)
    • An attached Audit Questionnaire (checklist or information request document) which can be used to organize the information and record transmittal
    • Requested due dates
    • Audit team members (with or without bio’s)
information requests guidelines
Information Requests Guidelines
  • Often it takes significant effort to the registered entity to collect and organize all of the information necessary for you to perform your sampling and testing.
  • All requests should be in writing and include:
    • Clear description of information requested
    • Date information should be received
    • Date information is received
    • Information request number with each line of requested information clearly identified with a unique number
  • All information requests should be logged initially and receipt of information recorded on the log.
clearly request information
Clearly Request Information
  • Your audit team should use a standard template for all information requests for additional information requested beyond the questionnaire.
information request log
Information Request Log
  • All data requested should be logged to track and protect registrant information. The following template shows the minimum information that should be recorded.
work paper preparation standards
Work Paper Preparation Standards
  • The notation “PV” means “physically inspected”
  • True False
  • Positive confirmations are always more reliable than negative confirmations because they require a member of the audit team to visit the confirming party’s location and interview them before we can confirm the quality of the evidence.
  • True False
  • ____ days prior to audit field work information should be requested from registrant. Can this info be requested via phone call or email?
  • True False
  • Work papers can be in an electronic format to be admissible in hearings
  • True False
work paper preparation and information requests
Work Paper Preparation and Information Requests
  • Questions
  • Section review
  • Follow-up items
module 6
Module 6

Audit Testing and Testing Methodologies

learning objectives4
Learning Objectives
  • What are the different kinds of sampling?
  • How do we do it?
  • Why sample?
audit testing and testing methodologies
Audit Testing and Testing Methodologies
  • Main tool in auditing is “sampling”
  • What is sampling?
  • Sampling is a systematic and defensible approach to drawing a conclusion of a population based on reviewing less than 100% of that population
audit testing and testing methodologies1
Audit Testing and Testing Methodologies

Why not sample 100% of the transactions?

  • Lack of time and resources
  • How do you know you have 100% of the population?
  • Finding “one more” negates validity of the entire sample
  • Rely on more on systematic controls in place in some cases for reliance that compliance is being met
sampling definitions
Sampling Definitions
  • Deviation Rate/Error Rate
    • Rate of acceptable errors in sample
  • Tolerable/Confidence Level
    • Rate of confidence that the auditor has that the sample selected is representative of the entire population
  • Population
    • Total number of items that can be tested
audit testing and testing methodologies2
Audit Testing and Testing Methodologies

Error Rate We’re Willing to Accept

audit testing and testing methodologies3
Audit Testing and Testing Methodologies

Generally accepted statistical sampling methods (SAS 111 – Amendment to Statement on Auditing Standards No. 39)

  • Haphazard
  • Systematic
  • Random
audit testing and testing methodologies4
Audit Testing and Testing Methodologies

Statistical sampling

  • The magic number is 45 for no errors at a 5% tolerable rate
  • The magic number is 77 for one error (helps determine the severity of the violation)
  • After that, stop the sample!
audit testing and testing methodologies5
Audit Testing and Testing Methodologies
  • Haphazard
    • Reach and grab approach; may be appropriate in certain situations
audit testing and testing methodologies6
Audit Testing and Testing Methodologies
  • Systematic
    • Select every nth item
    • Should be used in conjunction with statistical sampling
audit testing and testing methodologies7
Audit Testing and Testing Methodologies
  • Random
    • If items in population are identifiable by a number scheme, use Excel or other random number generation tool to determine items to be sampled
audit testing and testing methodologies8
Audit Testing and Testing Methodologies

Sample of Attributes Testing Work Paper

audit testing and testing methodologies9
Audit Testing and Testing Methodologies

Other Testing Methods

The 65% rule

The 100% rule

The “Test of One”

Testing to an insignificant level of remaining population, i.e.. 35% of the population

Expanding sample when an error is encountered

Stopping testing when an error is encountered – One and Done!

audit testing and testing methodologies10
Audit Testing and Testing Methodologies

What do you do when you find an error (potential violation in your sample)?

Increase sample size to the next level, if no additional errors (potential violations) = STOP!

If additional violations, consult enforcement staff….think about….

Requesting registrant to perform a 100% analysis or a larger subset of information to determine severity or pervasiveness of errors (potential violations)

Increasing the VSL of enforcement action (no additional sampling, document sampling result and include in work papers (i.e. record) for enforcement; additional testing by registrant can be included as part of a mitigation plan).

Example-one error (potential violation) found in sample of 65 is much different than finding four errors (potential violations) in the same sample size

NERC and Regional Entities should establish standardized procedures regarding sampling

audit testing and testing methodologies11
Audit Testing and Testing Methodologies


….thequalityof yourtestis dependent on the quality of the information you get for testing

….thedepthof yourtestis dependent on the depth of your information request

….thequalityof theevidenceyou gather is dependent on the cooperation you receive from the RE and their staff

…..thequalityof yourfindingsand recommendations are totally dependent on all of the above

audit testing and testing methodologies12
Audit Testing and Testing Methodologies
  • The main kinds of statistical sampling are:





  • The main advantage to sampling is:
  • ____________________________________________________________________________________________________________________________________
audit testing and testing methodologies13
Audit Testing and Testing Methodologies
  • 3. The initial sample size is always ___ items at a 95% confidence level
  • 4. If a piece of evidence is sampled that does not meet the assertion (standard or requirement) being tested we will sample another ____ items.
  • 5. If another piece of evidence is found in the sample that does not meet the assertion being tested what do we do?
  • _________________________________
  • It is never acceptable to have a sample size of one item of the population
  • True False
module 7
Module 7

Interview Techniques

learning objectives5
Learning Objectives
  • Techniques and tools used in the interview process
  • Standardized interview questions
  • Roles in the interview process
  • Meeting notices, interview records and logs
why conduct interviews
Why Conduct Interviews?
  • It’s a productive way of obtaining information
  • Opportunity to obtain information not found in documentation
  • Obtain off-record information
  • Meet knowledgeable & involved people and help us understand the organization being audited
  • Conduct interviews to supplement and/or clarify documentation
  • Verify procedures are being followed
how to interview effectively
How to Interview Effectively
  • Prepare list of questions in advance.
  • Start with introductory questions (name, title, organization, background)
  • Ask questions related to interviewee’s competence or area of expertise
    • A well prepared set of interview questions ensures your confidence in the completeness of information obtained efficiently and effectively
preparing for the interview
Preparing for the Interview
  • Identify need for interview and appropriate registered entity personnel
  • Call and set up interview. Be ready to describe the purpose of the interview
  • Prepare and send Interview Request form
  • Log the scheduled interview, date and purpose
  • Prepare written questions to ensure a complete and logical agenda for the meeting
  • Assign roles to attending audit team members
  • For ad hoc requested interviews, follow up with formal interview request
conducting the interview
Conducting the Interview
  • 1.Share the purpose of the audit
    • Objectives, goals, timing, confidentiality, and format
    • Audit teams’ roles and responsibilities
    • Explain what is expected from the registrant
  • 2. Use a sociable, conversational style
    • People will be more cooperative, less guarded
    • Avoid being harsh, grilling, or demanding
    • Be professional at all times
      • Avoid cursing
      • Don’t ask irrelevant questioning and discussions
  • 3. Let them know you will be taking minutes
  • 4. Use active listening
    • Polite responses
    • Ask for more….
    • Repeat back what you have heard
types of questions to ask
Types of Questions to Ask








  • Advantage
  • Generates explanations and descriptions
  • “ explain to me, describe, show me…”
  • Gets simple information
  • used tocheck facts, short answers, yes/no
  • Uncovers core issues
  • “How do you know...?
  • Ask when answers received contradict previous answers or observations
  • Check your understanding of what was said
  • “So let me see if I understand correctly…”
  • Ties together the main points covered in a session
leading questions
Leading Questions
  • Definition:
  • A leading question is one that suggests an answer, that implies there is a correct answer.
  • The term comes from law, where the courts insist that questions that suggest answers are not asked because they restrict the right of witnesses to speak freely.
use of leading questions
Use of Leading Questions
  • A leading question is a question that suggests the answer or contains the information the auditor is looking for.
  • For example:
  • “You were at Duffy\'s bar on the night of July 15, weren\'t you?”
  • (It suggests that the person was at Duffy\'s bar on the night in question).
  • The same question in a non-leading form would be:
  • “Where were you on the night of July 15?”
  • (This form of question does not suggest the answer the auditor hopes to elicit).
leading questions1
Leading Questions
  • Leading questions may often be answerable with a YES or NO, while non-leading questions are ‘open ended’. They also:
    • Point the respondent in the right way
    • Can be interpreted that the interviewer is not objective
  • Use of leading questions can be used where you are looking for information and the entity may use different naming criteria than what the auditor is used to. Use careful judgment in using leading questions.
more tips
More Tips…
  • Maintain eye contact
  • Be aware of body language
  • Take notes
    • It is often good for one person to ask the questions and have a second person to act as recorder
  • Talk little
  • Avoid answer-suggesting, accusatory and judgmental questions
  • Debrief after the interview – schedule a half hour between interviews to complete interview notes, use the restroom and get ready for the next interview
  • Explain the next steps in the process
in summary
In Summary…
  • Set a positive environment for the interviews
  • Use a conversational style during the interview
  • Maintain control of the interview
  • Use different types of questions during the interview
  • Use a variety of techniques for handling challenging interviews
interview techniques
Interview Techniques
  • Interview notes are ___ always ___ never admissible in compliance hearings.
  • Interview notes cannot ___ can ___ be used in reports in regards to findings of non-compliance.
  • Interview notes without a corroborating witness in the interview ____ are ___ are not considered audit evidence.
interview techniques1
Interview Techniques
  • 4. You have scheduled an interview with the Compliance Manager of the RE. Due to scheduling issues you are not able to do the interview during audit field work. The deadline for issuance of the report is fast approaching and this interview is not yet completed. What do you do?
  • ___ Rotate the interview until the next field audit
  • ___ Issue the audit report without doing the interview
  • ___ Hold off issuing the audit report until you can complete the interview
  • ___ Consider the lack of an interview a report finding
interview techniques2
Interview Techniques
  • 5. Interview requests, logs and notes should ____sometimes ___always be documented.
  • 6. It is ___ok ___not ok to rely on your memory in including interview comments in reports or hearings.
interview techniques3
Interview Techniques
  • Questions
  • Section review
  • Follow-up items
module 8
Module 8

Leveraging Project Management Tools & Techniques for Audit Success

learning objectives6
Learning Objectives
  • Project management audit tools and techniques
  • Why is budget important
  • Interview records and logs
  • Issue logs
  • Status reporting tools
  • Internal reporting issues log
  • Other dashboard tools

What is Project Management?





  • Project management is the process that will help us coordinate audits at a sufficient level of detail so that timely decisions can be made andrisks can be minimized.
  • This includes an incorporation of your knowledge, project management tools, skills, and techniques.
  • Projects have a beginning and an end.
  • Three phases of an audit:
  • 1. Planning
  • 2. Performance
  • 3. Reporting

Why Use a Disciplined Approach?

  • Project management will require you to focus on…
    • scope
    • cost
    • quality
    • effort
    • risk
    • timelines
  • … to ensure high quality service and audit success!
why do audits and projects fail
Why do audits and projects fail?
  • Lack of follow-through and accountability
  • Lack of time and focused resources
  • Loss of momentum or focus
  • Poor planning
  • Inability to transfer skills or knowledge to team or client
  • Lack of vision
  • Lack of cohesive leadership team
  • Inadequate emphasis on speed
  • Failure to allocate necessary resources
  • Organizational confusion and division
  • Lack of decision support information to measure success
  • Failure to maintain focus on the client
  • Errors in estimating how long tasks will take and what it will cost to do them

Project Management Life Cycle

Executing & Controlling: Managing the execution

of tasks, monitoring

progress and

determining what

needs to be changed




solutions &



the Project:

Evaluating success

and documenting

what was learned

Planning the


Determining the

steps to solve

the problem



  • Resolve operating problems
  • Conduct after action review
  • Reward & reassign team members
  • Define goals & objectives
  • Identify initial resources
  • Commit organization to goals
  • Develop work breakdown structure
  • Create project plan
  • Define targets
  • Maintain project status
  • Identify problems
  • Adjust targets & re-plan

* Planning occurs during this initial phase as well as through the rest of the project. As a manager it is essential to constantly plan and adjust the plan.


Planning is Important!

  • Planning answers questions such as:
  • What must be done?
  • Who will do it?
  • How will they do it?
  • How long will it take?
  • How much will it cost?

Our goal is to minimize risk

in our audit programs!


Execute and Manage the Plan

  • There are five main tasks during the execution phase. These include:
    • Maintain status
    • Conduct meetings
    • Adjust targets & re-plan
    • Manage resources
    • Communicate, communicate, communicate
follow the work breakdown structure
Follow the Work Breakdown Structure
  • A work breakdown structure is the logical sequence of work tasks and activities that are designed to deliver results.
  • In developing the work breakdown structure, five questions should be answered:
  • What tasks must be done?
  • Who will do each task?
  • How long will it take to do each task?
  • What materials/supplies are required?
  • How much will each task cost?

Identify Risks and Plan to Respond

  • Risks include anything that can stand in the way of meeting our goals and objectives.
  • Keep aware that real world events conspire to change our plans constantly. Stay alert to these changes and continuously work with your team leader revise your plans and communicate these changes to your audit team.

W. Edwards Deming has pointed out that, even when the fire department puts out a fire, you are no better off after the fire than you were before it. Fire prevention is far better than fire fighting.


Recognize Team Formation

A newly formed team needs structure to help them through four natural stages. Understand these stages are necessary, natural, and will eventually lead to great teamwork!

  • Performing
  • High morale and team identity forged by conceptual agreement and working patterns
  • Leader becomes team member
  • Forming
  • Conversations tend to stay at surface level
  • Heavy reliance on team leader
  • Storming
  • Conflict surfaces, usually around goals, roles and objectives
  • Need for strong problem solving skills
  • Norming
  • Members begin to agree on objectives and share feelings
  • Leader becomes more of a mentor and coach

Where do I Fit?

Outside influences directly affect your team. It is necessary to work through these to create a successful on-time audit that satisfies goals and objectives and is within budget.

Core Audit Team



  • A
  • Successful Audit

Completed On Time

Satisfies Goals and Objectives

Within Budget

Performed within CMEP rules



Decision Makers




Organizational Dynamics

Budgetary Constraints

Audit Team

Resource Constraints

Time Constraints

Key Stakeholders


General Process

  • A kick-off meeting should be held with each entity
    • Introduce project team and entity contacts
    • Discuss audit plan and requirements
    • Discuss information needed and communication protocols
  • Regular status meetings should be held with audit team and the registered entity
  • Status reporting is important to help keep to the schedule and the budget
  • Maintain interview and information request processes
  • Identify and resolve issues and conflicts and use escalation process when necessary
  • Focus on the plan

Tools – Status Reporting

Status Reports should be prepared on a regular basis in order to manage and maintain relevant information such as:

  • Accomplishments this Period
  • Activities Planned but not Accomplished
  • Activities Planned for next Period
  • Issues for Management Attention
  • Proposed Resolution to Issues
  • Action Items
  • Key Stakeholder Involvement
  • Important Communication Activities

Project Management

  • 1. Several project team members do not seem to get along with each other. As audit lead you should:

___ Bring it to their attention immediately that it is negatively impacting the team and try to find out more reasons why

___ Replace the less senior team member causing the issues

___ Tell them to go have a discussion and iron out their differences – “knock it off” and get along

___ Let things play out as long as the project runs smoothly and bring it up when the project is done in their evaluation


Project Management

  • As long as you have a detailed work plan regular project meetings are ___ necessary ___ not necessary to keep the project on time and on-track.
  • 3. The audit lead should ___ have ___ not have interim progress updates on the project with the registrant.
leveraging project management tools and techniques for audit success
Leveraging Project Management Tools and Techniques for Audit Success
  • Questions
  • Section review
  • Follow-up items
module 9
Module 9

Basic Report and Work Paper Review

learning objectives7
Learning Objectives
  • What do the standards require for work paper and report review?
  • What are tools that can make this process robust?
  • How do we link work papers to reporting for ease in future use in potential regulatory hearings?
  • What should the process be for work paper completion and review?
  • What should the process be for report preparation and review?
report and work paper review
Report and Work Paper Review
  • Project Close - File Review
    • File should bear evidence of review by supervisory staff
    • Review checklists are a necessary tool to ensure not only documentation requirements but to make sure nothing is missed
    • Review procedures should include make sure:
      • All replies to representation letters have been received and are properly dated and signed by the appropriate level of management
      • All checklists have been signed off and completed
      • Proper sign-offs have been made at the proper level
      • Staff evaluations and feedback have taken place
      • All pertinent points are moved to the file for the next audit
report and work paper review2
Report and Work Paper Review
  • File Review – What should we take out of the file?
    • “To-do” lists should be removed from the file prior to issuance
    • Review comments should be removed from the file
    • Superseded memos or information should not be included in the file
    • Always ask yourself, “This file is subject to legal discovery, what information is relevant to the report that we issued based on the evidence we collected and examined?”
report and work paper review3
Report and Work Paper Review
  • Quality Assurance – File Control
    • Lock out file access after a time certain period-prevent the potential for manipulation of findings
    • Retention periods – follow organizational directives and CMEP rules – should be the longer of five years or audit cycle for retention
report and work paper review4
Report and Work Paper Review
  • Work Paper Linkage
  • There must be linkage between every report finding back to an evidential work paper in order for it to be a credible and defensible finding
  • Think “if I were on the stand how would I document and defend this finding?”
  • Leave a trail you can find in the future to take you back to your thinking at this point in time….leave some breadcrumbs
report and work paper review and quality control
Report and Work Paper Review and Quality Control

File Review, Report Preparation and Issuance Process

report and work paper review9
Report and Work Paper Review

File Review, Report Preparation and Issuance Process

report and work paper review10
Report and Work Paper Review
  • File Review and Reporting
    • Audit standard is that a professional not involved with the audit must be able to reach the same conclusion as the auditor – using the same information
      • Does the file you’re reviewing meet this standard?
    • File evidence must meet the requirements of “Audit Documentation”
    • Report should not be issued in final until the file has been reviewed and all sign-offs and quality assurance procedures have taken place
    • Hard to teach “quality assurance” for report review – get your most thorough and inquisitive reviewer on it for the best results
    • Utilize an internal inspection process to keep file issuance robust and meeting standards
report and work paper review11
Report and Work Paper Review
  • Quality assurance reviews of your report are not necessary because only your audit team members can interpret the audit evidence to come to the conclusions in the report.
  • True False
  • 2. In a regulatory hearing regarding your report findings your recollection of interviews
  • will
  • will not suffice as supportable evidence.
report and work paper review12
Report and Work Paper Review
  • 3.Once we have reviewed a compliance area and found no issues the next audit should
  • Exclude
  • Include the same area for review.
  • 4. Audit files should
  • Never be shredded
  • Shredded in accordance with organizational standards.
report and work paper review13
Report and Work Paper Review
  • Questions?
  • Follow-up items
module 10
Module 10

Conflict Resolution and Escalation Protocols

learning objectives8
Learning Objectives
  • Conflicts are natural and expected
  • How to handle issues and conflicts
  • How and when to escalate due to issues or lack of performance
  • Focus on resolution and moving forward
successful audits
Successful Audits…
  • Successful management of audits always requires informed, proactive and timely management of issues.
  • Conflicts and Issues:
    • Are resolvable with action items
    • Can be escalated
    • Are proactively discovered during the course of the audit
    • Your leadership team will need to analyze a myriad of concerns and issues
      • Audit scope
      • Shortage of resource issues
      • Differences in interpretation, intent and styles
  • Good issue management and escalation processes should result in timely resolution of issues and conflicts!
different conflicts treat differently
Different Conflicts Treat Differently
  • Conflicts related to the conduct of audits can be escalated for discussions through an informal process using senior management.

Conflicts or disagreements with findings (potential violations) or not providing necessary information are not subject to the same escalation procedures (the CMEP has escalation and due process procedures which the Regional Entity staff person should reference).

issue management
Issue Management
  • The issue management process involves monitoring the status of each of the concern, issue or conflict. The issue management process is depicted as follows:

Clarification &













Check for





Involve all




Specify scope


Document progress

Assign owner

Find solution

Set due date

Plan implementation

issue management1
Issue Management
  • All issues should be logged and reviewed with the Audit team.

Most issues will be able to be resolved within the team. However, if the issues are beyond the control of the team, or impede the progress of the audit, it may be necessary to use anescalationprocess.

escalation is sometimes needed
Escalation is Sometimes Needed
  • Sometimes issues will need to be escalated to get resolution
  • Reasons for escalation are:
    • If the issue is ‘Mission Critical’
    • Issues beyond the authority of an individual or team and require a consensus decision
    • Owner of the issue is not clear or cannot be established
    • Issues which are not being properly, addressed and may impede the progress of the audit (will affect the audit scope, costs and/or timeline)
  • Your leadership team will evaluate the issue and determine the appropriate steps to ensure that the issue is resolved.
why escalate
Why Escalate?
  • Potential events requiring escalation
  • Missed deadline by 24 hours
  • Lack of meeting or interview attendance with no notice
  • Lack of access to necessary IT systems
  • Interview or meeting does not occur within 5 business days of original agreed to date
  • Refusal to supply information
  • Incomplete information
  • Lack of returning calls after initial call and follow up call
  • Remind the registrant that you must make a determination of compliance and failure to produce sufficient evidence to make such a determination will result in an alleged violation
what is required
What is Required?
  • All information and meeting requests (including interviews) will be documented in a log by requesting individual.

Each corresponding escalation will be documented in a log which will be made available to project management and the registrant on a regular basis. The documentation should include:

      • Requested information or meeting
      • Requestor
      • Registrant individual contacted
      • Response date or date information received
      • Follow-up dates
      • Escalation protocol steps
don t forget
Don’t Forget
  • Keep to the facts
  • Assume the best
  • Involve your team to reach the best solution
  • Conflicts and issues are a natural part of the process – don’t look away!
  • Timely resolution of issues is needed so that they don’t fester and create new issues
  • Registrants can always use the due process protections of the CMEP
  • Once a decision is made or a conflict resolved, move forward!
conflict resolution
Conflict Resolution
  • The registrant’s Compliance Manager comes to the audit lead and says an audit team member is “too nosy and pushy” in performing the audit. The audit lead should:

___ replace the team member

___ congratulate them on a job well done

___ coach them to make sure their approach is collaborative yet gets the audit evidence needed

conflict resolution1
Conflict Resolution
  • 2. The registrant’s Compliance Manager is a “good guy” yet constantly misses deadlines in getting information to the team. The audit team members should
  • ___ cut him some slack in meeting deadlines while encouraging him to get them the information
  • ___ document these issues and follow the escalation protocol
  • Any instances involving continuous use of the escalation protocols
  • ___ should
  • ___ should not be part of report findings
conflict resolution and escalation protocols
Conflict Resolution and Escalation Protocols
  • Questions
  • Section review
  • Follow-up items
module 11
Module 11

Audit Close Activities, Lessons Learned and On-going Performance Management

learning objectives9
Learning Objectives
  • Why is closeout important?
  • What is an exit meeting?
  • Why collect and discuss Lessons Learned?
  • Is performance management an annual process?
  • What are some possible audit performance measures?
why a closeout phase
Why a Closeout Phase?
  • It is important to ensure that a project is properly closed for two reasons:
    • First, there is a tendency for projects to drift on and on, and never end.
    • Secondly, it is important to ensure that the work of the audit team is acknowledged and that the lessons to be learned from the project are formally investigated and recorded for use on the next project.
  • Lack of closure leaves everyone feeling dissatisfied and unrewarded for the work (often extra work) that they have done.
the closing phase
The Closing Phase
  • The purpose of the Closing Phase is to formalize the acceptance of the final audit results, share knowledge gained during the project and bring the project to an orderly end.

Key Steps include:

    • File and/or return to the registered entity all working or final documents
    • Complete internal final Quality Assurance Review
    • Have the team complete their self-evaluation forms
    • Hold Close-Out Meetings with the registered entity and with the project team. Lessons Learned from each meeting should be documented and shared with the project team.
    • Ensure all outstanding issues or tasks that were identified prior to and during the Close-Out Meetings have been resolved or a course of resolution has been determined.
    • Complete the performance review process for all audit team members
    • Lockout the audit file
    • Organize and hold a closing celebration for the audit team to celebrate the success of the project
exit meeting with registrant
Exit Meeting with Registrant
  • Your closing presentation at the exit meeting should include:
  • Preliminary audit findings, with strong emphasis on the factual basis for each compliance or noncompliance item identified by the audit team, review CMEP due process protections and next steps for any findings (alleged violations),
  • Solicit feedback on how the audit was performed and any closing comments from the entity. Be sure to give the audited entity an opportunity to voice any challenges to facts as understood by the team. This should be reinforced when the audited entity has a draft report to review. Also, provide the audited entity with the opportunity to review the conclusions and supporting factual basis.
  • If it is your organization’s policy - present informal recommendations for improvements verbally related to your observations during the audit
lessons learned for the audit team internal
Lessons Learned for the Audit Team (Internal)
  • When the audit is complete, the team should hold a review meeting to discuss the audit and identify…
  • What went well
  • What did not go well
  • What to replicate in future audits
  • What to change
  • Without assigning blame, the members conduct the audit review with the goals of learning from mistakes and improving future projects. Often called a postmortem, the audit-review meeting sometimes takes place shortly before the end of the project rather than afterward, because team members often must leave the project shortly before it ends.
  • The team documents suggestions for change as action items in the next audit plan.
lessons learned continued
Lessons Learned (continued)
  • Allow the team time to reflect on and prepare for the review.
    • Ensure there is plenty of time for group discussion.
    • Keep the meeting positive
    • “Doing better next time” is the theme of the review
    • Document and share the results of the meeting
  • Be sure to celebrate in some way the end of the audit – we all want to be appreciated!

Electric Reliability Organization

Regional Entity Management Group

on going audit performance management cycle
On-going Audit Performance Management Cycle

Review Audit Plans

“What do we want to achieve?”

“Set SMART goals – specific, measurable, achievable, realistic and timely goals

Set Performance Goals


Meet and Discuss expectations

“What are growth areas and needed skills?”

MonitorandEvaluate Performance

“How much progress are we making toward our goals?”

is this just an annual process
Is This Just an Annual Process?
  • NO!
  • It is important to routinely throughout the year discuss performance. It is also important to discuss performance on each (or larger) audits.
  • Be sure to….
    • Establish performance goals and expectations for a specified period of time during an audit.
    • Provide feedback regarding the performance to the stated goals and expectations as well as to the standards of good, solid performance at a particular level of responsibility.
    • Establish specific actions required to improve or demonstrate performance in specific areas.
    • Reinforce employees strengths that should be recognized and maintained.
possible measures
Possible Measures
  • Quantitative measures might include:
  • Plan completion:This is a measure at a point in time. This may be measured using the number of projects completed, weighted by the planned size of each project, with estimates for projects in-progress. Measured throughout the year, it is compared against the percentage of the year elapsed.
  • Report issuance:This is a measure of the time elapsed from completion of testing to issuance of the final audit report, including management’s action plans. This can be measured in average days or percentage of reports issued within a certain standard, such as 30 days.
  • Issue closure:The number of days that reported issues remain open, or open after their agreed-upon closure date, are key measures.
  • Staff utilization rate:This is measured as the percentage of time spent on audit, as opposed to administrative time such as training or vacation.
retain your team
Retain Your Team

Developing and retaining quality professionals is a key concern!

  • Key methods for developing and retaining internal audit staff personnel include:
    • Provide challenging, varied assignments
    • Ensure great supervision
    • Have staff participate in audits from start to finish, to learn all phases of the audit process
    • Provide opportunities to lead (in-charge) audits, starting with smaller, easier audits
    • Involve staff in improvement task forces, such as preparation for quality assurance review
    • Have them participate in the recruiting and interviewing process for new hires
    • Rotating through various audit teams
    • Provide both outside training (e.g., seminars) and in-house training for two weeks/year
audit close process
Audit Close Process
  • The audit file should always be open for audit findings. One never knows when instances of non-compliance will be communicated

True False

  • An audit debrief meeting should be held just between the audit lead and organization upper management
  • True False
audit close process1
Audit Close Process
  • Goals should not be established up front before the audit for audit team staff.
  • True False
  • 4. Audit project lead time is better spent interacting with the registrant vs. assisting staff in executing the work plan.
  • True False
  • 5. Audit staff should be capable of reading and understanding the audit work plan and executing the audit steps without a great deal of supervision or questions.
  • True False

Regional Entity Management Group

audit close process2
Audit Close Process
  • Questions
  • Section review
  • Follow-up items