Phil o donnell manager operations planning audit team
This presentation is the property of its rightful owner.
Sponsored Links
1 / 87

Phil O’Donnell Manager, Operations & Planning Audit Team PowerPoint PPT Presentation


  • 108 Views
  • Uploaded on
  • Presentation posted in: General

Phil O’Donnell Manager, Operations & Planning Audit Team. Manager, Operations & Planning Audit Teams CFE Standards and Requirements Presenting Best Evidence January 11, 2012. Objectives. Present the CFE Approved Standards for 2012 Identify the parts of the Standard

Download Presentation

Phil O’Donnell Manager, Operations & Planning Audit Team

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Phil o donnell manager operations planning audit team

Phil O’DonnellManager, Operations & Planning Audit Team

Manager, Operations & Planning Audit Teams

CFE Standards and RequirementsPresenting Best EvidenceJanuary 11, 2012


Objectives

Objectives

  • Present the CFE Approved Standards for 2012

  • Identify the parts of the Standard

  • Provide guidance on maintaining good evidence and documentation.

  • Identify pending standards for 2012.


Elements of every standard

Elements of Every Standard

  • Title

  • Number

  • Purpose

  • Applicability

  • Effective Date

  • Requirements

  • Measures

    • Compliance Monitoring Responsibility

    • Compliance Monitoring Period and Reset Timeframe

    • Data Retention


Bal 001 mx 0 real power balancing control performance

BAL-001-MX-0 Real Power Balancing Control Performance

Propósito:

  • Mantener en estado estable la frecuencia de la Interconexión, dentro de límites definidos mediante el balance de la demanda de energía y la generación en tiempo real.

  • Aplicabilidad:

  • Autoridades de Balance

  • Área de Control Baja California (ACBC).


Bal 001 mx 0 real power balancing control performance1

BAL-001-MX-0 Real Power Balancing Control Performance

FechaEfectiva:

1 de Enero del 2012.

The date compliance is expected


Bal 001 mx 0 real power balancing control performance2

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 1

Cada Autoridad de Balance deberá operar de tal manera que, en un base rolada de 12 meses, el promedio de los promedios minuto-reloj del Error de Control de Área (ECA) de la Autoridad de Balance dividido entre 10B (B es el promedio de minutos-reloj del Bias de la Frecuencia de la Autoridad de Balance) multiplicado por los promedios minutos-reloj correspondientes al Error de Frecuencia de la Interconexión, sea menor que un límite específico. Este límite Ɛ12 es una constante derivada de una frecuencia objetivo establecida (calculada por separado para cada Interconexión) la cual el Comité Operativo de NERC revisa y ajusta como sea necesario.


Bal 001 mx 0 real power balancing control performance3

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 1


Bal 001 mx 0 real power balancing control performance4

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 1

The requirement specifies and defines how ACE is calculated.

  • ACE = ECA Error de Control de Área

    ECA = (NIA - NIS) – 10B (FA - FS) – IME

  • Definition of each term is specified in the standard


Bal 001 mx 0 real power balancing control performance5

BAL-001-MX-0 Real Power Balancing Control Performance

Ɛ1 For The Western Interconnection


Bal 001 mx 0 real power balancing control performance6

BAL-001-MX-0 Real Power Balancing Control Performance

Frequency Bias (B)


Bal 001 mx 0 real power balancing control performance7

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 1

Very complicated wording, but essentially the requirement is to maintain your Area Control Error (ECA) So that the average over a long term (12 month period) remains within a certain range.


Bal 001 mx 0 real power balancing control performance8

BAL-001-MX-0 Real Power Balancing Control Performance

Measurement 1

  • Provides a description of how you determine if you are in compliance

  • Describes in detail how the elements in R1 are calculated to establish a percentage value CPS1

  • This CPS1 value is determined each month based on the previous 12 months of data and must be greater than or equal to 100% to be compliant with R1.


Bal 001 mx 0 real power balancing control performance9

BAL-001-MX-0 Real Power Balancing Control Performance

FC Minutos-reloj

FC Horas-reloj

FC Horas-reloj en mespromedio

FC mes

FC 12-meses


Bal 001 mx 0 real power balancing control performance10

BAL-001-MX-0 Real Power Balancing Control Performance

  • The FC 12-meses value is used to determine an overall Compliance Factor

Finally, a CPS1 value is determined by:

CPS1 = (2-FC) x 100%


Bal 001 mx 0 real power balancing control performance11

BAL-001-MX-0 Real Power Balancing Control Performance

  • Each BA is required to report its CPS1 values each month.

  • These are reviewed monthly by a Compliance Auditor


Bal 001 mx 0 real power balancing control performance12

BAL-001-MX-0 Real Power Balancing Control Performance

What are we looking for when we audit.

Compliance Assessment Approach for R1

  • Review CPS1 logs, (the monthly reports)

  • Identify calculation used to calculate CPS1

  • Identify the target frequency bound; ɛ 1

  • Identify the Frequency Bias Setting


Bal 001 mx 0 real power balancing control performance13

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 2

  • Cada Autoridad de Balance deberá operar de tal manera que su ECA promedio esté dentro de un límite específico L10 por al menos en el 90% de los períodos de diez minutos-reloj (6 períodos que no se traslapen, por hora) durante un mes calendario.

    PROM10-minutos (ECAi) ≤ L10


Bal 001 mx 0 real power balancing control performance14

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 2

This requirement is to maintain your Area Control Error (ECA) So that the it does not become too large over a 10 minute period.


Bal 001 mx 0 real power balancing control performance15

BAL-001-MX-0 Real Power Balancing Control Performance

Frequency Bias (B)


Bal 001 mx 0 real power balancing control performance16

BAL-001-MX-0 Real Power Balancing Control Performance

Measurement 2

  • Provides a description of how you determine if you are in compliance

  • Describes in detail how the elements in R2 are calculated to establish a percentage value CPS2

  • This CPS2 value is determined each month based on performance for the previous month and must be greater than or equal to 90% to be compliant with R2.


Bal 001 mx 0 real power balancing control performance17

BAL-001-MX-0 Real Power Balancing Control Performance

Measurement 2

  • The measurement describes how you use data to determine if during any 10 minute period you incur a “violation”


Bal 001 mx 0 real power balancing control performance18

BAL-001-MX-0 Real Power Balancing Control Performance

Measurement 2

  • Cada Autoridad de Balance deberá lograr, como mínimo, el 90% de cumplimiento con el Requerimiento R2 (CPS2). El CPS2 se refiere a un límite en el promedio de 10 minutos del ECA. Un porcentaje de cumplimiento se calcula como sigue:


Bal 001 mx 0 real power balancing control performance19

BAL-001-MX-0 Real Power Balancing Control Performance

  • Each BA is required to report its CPS2 values each month.

  • These are reviewed monthly by a Compliance Auditor


Bal 001 mx 0 real power balancing control performance20

BAL-001-MX-0 Real Power Balancing Control Performance

What are we looking for when we audit.

Compliance Assessment Approach for R2

  • Review monthly CPS2 logs for past 12 months

  • Verify the average ACE was within the BAs L10 for the last 12 months for at least 90% of the time periods monitored.


Bal 001 mx 0 real power balancing control performance21

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 3

Cada Autoridad de Balance que proporciona Servicio de Regulación de Traslape, deberá evaluar el Requerimiento R1 (por ejemplo, el Estándar de Desempeño de Control 1 o CPS1) y el Requerimiento R2 (por ejemplo, el Estándar de Desempeño de Control 2 o CPS2); utilizando las características del ECA combinado y los Ajustes del Bias de la Frecuencia combinados.


Bal 001 mx 0 real power balancing control performance22

BAL-001-MX-0 Real Power Balancing Control Performance

Requirement 4

A cualquier Autoridad de Balance que recibe Servicio de Regulación Traslape, no se le evaluará su desempeño de control (por ejemplo, desde una perspectiva de desempeño de control, la Autoridad de Balance ha turnado todos los requisitos a la Autoridad de Balance que proporciona el Servicio de Regulación Traslape).


Bal 001 mx 0 real power balancing control performance23

BAL-001-MX-0 Real Power Balancing Control Performance

  • There are no specific Measures defined for R3 and R4

  • If providing Overlap Regulation you would be assessed as in R1 and R2.

  • If receiving Overlap Regulation Service we would verify this and perform no other assessment.


Bal 001 mx 0 real power balancing control performance24

BAL-001-MX-0 Real Power Balancing Control Performance

  • Compliance Monitoring Period and Reset Timeframe

    • One Calendar Month

  • Data Retention Period

    • One Year


Bal 001 mx 0 real power balancing control performance25

BAL-001-MX-0 Real Power Balancing Control Performance

Data to Retain

  • ECA promedio de un minuto (ECAi)

  • Error de Frecuencia promedio de un minuto

  • si se está utilizando un Bias variable, el Bias de Frecuencia promedio de un minuto.


Int 001 mx 0 interchange information

INT-001-MX-0 InterchangeInformation

Propósito:

Asegurar que la información de Intercambio de Energía se envíe al servicio de análisis de confiabilidad aprobado por NERC.

  • Aplicabilidad:

  • Entidades de Compra-Venta.

  • Autoridades de Balance.


Int 001 mx 0 interchange information1

INT-001-MX-0 InterchangeInformation

FechaEfectiva:

1 de Enero del 2012.


Int 001 mx 0 interchange information2

INT-001-MX-0 InterchangeInformation

Requirement 1

  • La Entidad de Compra-Venta y la que Sirve la Carga, deberán asegurar que se envíe el Intercambio Acordado a la Autoridad de Intercambio para Todos los Programas Dinámicos en el perfil esperado de MW promedio por cada hora..


Int 001 mx 0 interchange information3

INT-001-MX-0 InterchangeInformation

Measure 1

La Entidad de Compra-Venta que sirve la carga deberá tener y proporcionar a solicitud evidencia que pueda incluir pero no se limita a, sus relatorios de operación de etiquetas de Transacción de Intercambio, grabaciones de voz o transcripciones de grabaciones de voz, comunicaciones electrónicas, impresiones de computadoras u otra evidencia equivalente que será utilizada para confirmar que el Intercambio Acordado se envió a la Autoridad de Intercambio para todos los Programas Dinámicos del perfil de MW promedio esperado por hora, como se especifica en el Requerimiento 1.


Int 001 mx 0 interchange information4

INT-001-MX-0 InterchangeInformation

Requirement 2

  • La Autoridad de Balance Receptora deberá asegurar que el Intercambio Acordado se envíe a la Autoridad de Intercambio:

    • R2.1. Si la Entidad de Compra-Venta no está involucrada en el Intercambio, como la entrega desde un generador con propietario en conjunto.

    • R2.2. Para cadapago bilateral de IntercambioInadvertido.


Int 001 mx 0 interchange information5

INT-001-MX-0 InterchangeInformation

Measure 2

Cada Autoridad de Balance Receptora deberá tener y proporcionar a solicitud, evidencia que puede incluir pero no se limite a, sus relatorios de operación de etiquetas de Transacción de Intercambio, grabaciones de voz o transcripciones de grabaciones de voz, comunicaciones electrónicas, impresiones de computadora u otra evidencia equivalente que se utilizará para confirmar que el Intercambio Acordado se envió a la Autoridad de Intercambio como se especifica en los Requerimientos 2.1 y 2.2.


Int 001 mx 0 interchange information6

INT-001-MX-0 InterchangeInformation

  • The Interchange Authority (Autoridad de Intercambio) is the Electronic Tag Authority provided by WECC.

  • Submitting an e-tag for the dynamic schedule satisfies this requirement.


Int 001 mx 0 interchange information7

INT-001-MX-0 InterchangeInformation

What are we looking for when we audit.

Compliance Assessment Approach for R1

  • Determine if BA uses Dynamic Schedules.

  • If so verify that e-tags have been submitted for all Dynamic Schedules at the expected average MW profile for each hour.


Int 001 mx 0 interchange information8

INT-001-MX-0 InterchangeInformation

What are we looking for when we audit.

Compliance Assessment Approach for R2

  • Determine if BA has received any interchange transactions that do not involve a PSE.

  • Determine if BA has received any bilateral inadvertent payback from another BA

  • Verify the existence of an e-tag if either condition above is true.


Int 001 mx 0 interchange information9

INT-001-MX-0 InterchangeInformation

  • Compliance Monitoring Period and Reset Timeframe

    • 12 months from the last finding of non-compliance

  • Data Retention Period

    • 90 days

    • If a violation is found, Current plus 2 years.


Bal 001 mx 0 real power balancing control performance26

BAL-001-MX-0 Real Power Balancing Control Performance

Data to Retain

  • Electronic copies of Interchange Transaction Tags showing time submitted and MW profile.

  • Records of actual MW profiles of any Dynamic Schedules.


Presenting evidence

Presenting Evidence

Auditors have two primary concerns when reviewing documentation or evidence to determine compliance with a standard.

  • Is the evidence sufficient to establish compliance?

  • Is the evidence of adequate quality to establish compliance?


Presenting evidence1

Presenting Evidence

  • Evidence and Documentation expectations were not well defined when the compliance program was first started in the US.

  • Poor quality or insufficient documentation was a significant cause of possible violations

  • WECC auditors (and the Registered Entities) have learned a lot since 2007.

  • This presentation is intended to make you aware of evidence principles and avoid issues at the start.


Compliance audits

Compliance Audits

Compliance Audits are considered “Performance Audits”

  • Performance audits are defined as engagements that provide assurance or conclusions based on an evaluation of sufficient, appropriate evidence against stated criteria, such as specific requirements, measures, or defined business practices.


Audit objectives determine required evidence

Audit Objectives Determine Required Evidence

  • Audit Objectives define required scope and outcomes.

  • The scope and possible outcomes impact evidence needs.

  • Consideration of Audit Risk will also impact evidence requirements.

  • Evidence needs for Audit may be different than for Spot Check or Mitigation Plan Closure.

  • Evidence needs for Mitigation Plan Closure may vary depending on reason for Mitigation Plan.


Audit risk

Audit Risk

  • No Auditor is perfect

  • There is risk that the matter under audit may contain an inaccuracy

  • There is risk that evidence may be misinterpreted

  • The greater the risk the greater the quantity and quality of evidence required


Sufficiency and appropriateness

Sufficiency and Appropriateness

Sufficiency of Evidence

  • The measure of the quantity of evidence

  • Quantity of evidence is dependent on the scope of the audit

  • Quantity of evidence is dependent on the audit risk


Phil o donnell manager operations planning audit team

Quantity does not make up for poor quality!


Sufficiency and appropriateness1

Sufficiency and Appropriateness

The Number one problem we experience with respect to Evidence Sufficiency

  • Evidence not provided or available for entire audit period!

    Other issues

    • Large numbers of source documents with poor organization (minimal or no summaries or roadmap)


Sufficiency and appropriateness2

Sufficiency and Appropriateness

Data Retention

  • Specified in some standards

  • If not specified in the standard the default audit period is from the time of the Audit back to the later of:

    • Date of Previous Audit

    • Date of Registration (or designation by CENACE)

    • Effective Date of Standard (January 1, 2012)


Sufficiency and appropriateness3

Sufficiency and Appropriateness

Sampling is used to limit the amount of detailed evidence provided.

  • Normally used in conjunction with summary of a full set of data.

  • Sampling used to assess details.

  • Reduces the burden on the Audit Team but not really on the Entity

  • Audit Team must select the samples


Sufficiency and appropriateness4

Sufficiency and Appropriateness

Appropriateness

The measure of the quality of evidence

  • Relevance

  • Validity

  • Reliability


Phil o donnell manager operations planning audit team

Everything we see may not be as it appears


Sufficiency and appropriateness5

Sufficiency and Appropriateness

Quality of Evidence

  • Good Internal Controls point to reliable evidence

  • Direct observation is more reliable than indirect observation

  • Examination of original documents is more reliable than examination of copies

  • Testimonial evidence is considered more reliable when person can speak freely.

  • Testimonial evidence from system experts is more reliable than from personnel with indirect or partial knowledge


Sufficiency and appropriateness6

Sufficiency and Appropriateness

Quality of Evidence

  • One indicator of quality is consistency of format of the same evidence type.

    • Procedures should have the same format

    • Data Summaries such as spreadsheets should have similar format or characteristics

      We understand there are some limitations to this but it is an indicator.


Sufficiency and appropriateness7

Sufficiency and Appropriateness

Another indication of Quality of evidence is the “Roadmap” or your instructions to the audit team about where specific evidence is located.

  • Although not a part of the evidence itself it is an indicator of your company’s process.

  • Accurate indexing and references add a level of reliability to your overall documentation process.


Types of evidence

Types of Evidence

  • Physical Evidence

  • Documentary Evidence

  • Testimonial Evidence

    Compliance Audits may use all three types but Documentary Evidence is by far the most frequent type of evidence assessed.


Types of evidence1

Types of Evidence

Physical Evidence

  • Physical evidence is obtained by auditors’ direct inspection or observation of people, property, or events. Such evidence may be documented in summary memos, photographs, videos, drawings, charts, maps, or physical samples.

    • Used during Control Room visits

    • Direct Observation of Operator Tools and Facilities


Types of evidence2

Types of Evidence

Documentary Evidence

  • Documentary Evidence is obtained in the form of existing information such as Policies, Procedures, Programs, Logs, Records, Transcripts, letters, emails, contracts, invoices, spreadsheets, database extracts, electronically stored information, and management information on performance.

  • Most of what we review falls into this category


Types of evidence3

Types of Evidence

Testimonial Evidence

  • Testimonial evidence is obtained primarily through inquiries, interviews, and questionnaires

    Auditors assess a lot of testimonial evidence but a great deal of it is used as “Corroborating” Evidence to support Documentary Evidence or Observations.


Types of evidence4

Types of Evidence

Corroborating or Stacking Evidence

  • Redundant evidence which backs up other evidence

  • Used to increase the reliability and validity of evidence

    • Alternate Source of evidence

      • Attestation of 3rd party to back up your Attestation

      • Logs to back up reports

      • Maintenance Records to Back up Maintenance Summaries

    • Alternate Type of evidence

      • Physical to back up documentary

      • Documentary to back up Attestation

    • Testing (computation or verification of calculations)


Examples of evidence

Examples of Evidence

  • We have talked in general about:

    • Sufficient Evidence

    • Appropriate Evidence

    • Types of Evidence

  • Let’s Look at some examples…


Evidence examples

Evidence Examples

Standards which require Policies, Procedures, Plans, Programs,

Contain words in the standards such as:

  • …shall specify its Contingency Reserve policies

  • …shall have procedures

  • …Shall have written operating instructions and procedures

  • …needs to develop, maintain, and implement a set of plans

  • …plans shall include:

  • …shall have a training program

  • …shall document, maintain, and publish facility connection requirements


Quality evidence for procedural documents

Quality Evidence for Procedural Documents

The characteristics of a valid procedural or policy document include:

  • Document title

  • Definition or Purpose

  • Revision level

  • Effective date

  • Authorizing signatures


Sufficient evidence for procedural documents

Sufficient Evidence for Procedural Documents

Document History for the entire audit period.

  • All previous versions should be available in accordance with the data retention specifications

  • Documents should have a “Revision History”


Sufficient evidence for procedural documents1

Sufficient Evidence for Procedural Documents

Characteristics of good revision history

  • Version or Revision ID

  • Date of Revision

  • Reason for Revision


Sufficient evidence for procedural documents2

Sufficient Evidence for Procedural Documents

Examples of Good Revision Changes:

“ Minor errata changes to correct typos”

“ Revision to include lessons learned from annual emergency drill”

“Annual Review performed”

“Revision to incorporate new Standard Requirements”

“Revision to new procedure format, no content changes”

  • Revision History should go back to the first version of the document.


Sufficient evidence for procedural documents3

Sufficient Evidence for Procedural Documents

HINT:

Avoid misinterpretation, try and use terminology that is used in the standards.

  • If Standard calls for a “plan” consider calling your Document a “plan”

  • If Standard calls for a “protocol” consider calling your document a “protocol”


Evidence examples1

Evidence Examples

Standards which require performing actions,

Contain words or phrases such as

  • shall perform hourly error checks

  • Transmission Operator shall demonstrate, through simulation or testing.

  • Transmission Operator shall operate within…

  • shall take immediate steps…

  • shall periodically conduct and document an assessment

  • Transmission Operator shall notify…

  • shall staff all operating positions with personnel that are NERC certified.


Evidence examples2

Evidence Examples

  • When the standard calls for a task to be performed it must be documented.

  • Evidence of performance of tasks include

    • Records

    • Logs

    • Reports

    • Work Orders

    • Phone recordings

    • Transcripts of phone recordings

    • Shift Schedules


Quality indicators of performance evidence

Quality indicators of Performance Evidence

  • Controls are established for documenting performance

  • Original documents available

  • Documents contain details of performance and statement of completion of task

    • What was done

    • Who did it

    • When it was done

    • Why was it done

  • Some type of review or approval


Sufficiency of performance evidence

Sufficiency of Performance Evidence

  • Records available for all instances of performance.

  • Frequent and repeat performance have summary documents.


Quality indicators for data

Quality Indicators for Data

  • Process is identified for regular capture of data

  • Process and methodology is specified for calculated parameters

  • Explanation of data records available

  • Calculation can be verified.

  • Data does not have significant gaps (lost data)

  • Consistency of Format


Evidence examples3

Evidence Examples

Data that is required by Standards

  • CPS1 Value and Calculation (BAL-001-MX-0)

  • CPS2 Value and Calculation (BAL-001-MX-0)

  • DCS Value and Calculation

  • ACE Value and Calculation (BAL-001-MX-0)

  • Disturbance Recovery Period ACE recovers within

  • Frequency Bias Settings Value and Calculation

  • AGC Mode, when and how long in each mode

  • ATEC in service, When and When Not in this mode

  • Voltage Regulator in Auto and Controlling Voltage

  • PSS in Service

  • Inadvertent Interchange

  • Transaction Tags and Approval History (INT-001-MX-0)


Sufficiency indicators for data

Sufficiency Indicators for Data

  • Required Data is available for entire audit period or as specified in standard.

  • Supporting data used for calculation is also available for same period


Challenging or special evidence situations

Challenging or Special Evidence Situations

  • Documenting Negatives

  • Evidence of “Coordination”

  • Evidence of “Distribution”

  • Subjective Requirements

  • Training Records


Documenting negatives

Documenting Negatives

  • Event Driven Standards

  • Non-Applicability

  • Do not own Equipment

  • Primary evidence for these is attestation

  • A quality attestation includes some basis.

    …Based on a review of the logs/daily reports

    …Based on a review of the maintenance records

  • Attestation from 3rd party when possible is great for corroboration


Evidence of coordination with other entities

Evidence of “Coordination” with other entities

  • Typical evidence provided initially is a single email.

    “…If you have any comments please contact ______”

    This alone is neither sufficient or appropriate to demonstrate coordination between one or more parties.

  • If emails or correspondence are used

    • Two way communications are suggested

  • More appropriate are:

    • Meeting Agendas

    • Meeting Minutes

    • Attendance Lists


Evidence of distribution of information

Evidence of “Distribution” of information

  • Typical evidence provided initially is a single email with a large distribution list.

    “…please see attached”

    This alone is typically neither sufficient or appropriate to demonstrate distribution to others.

  • If emails or correspondence are used

    • Need clear identification of the personnel on the distribution list.

  • Even Better is corroboration by receipt acknowledgement


Subjective requirements

Subjective Requirements

  • Requirements which are not specific

  • Provide options for the entity

  • Open Ended

    • As applicable

    • Where necessary

    • Where appropriate

    • Consider the following

  • Flexibility intended to address a variety of organizations.

  • Professional audit guidance indicates that the audited organization must determine the details of what is appropriate or applicable to their organization within the context of subjective requirements….BUT


Subjective requirements1

Subjective Requirements

  • Auditors must ensure the subjective elements are correct and necessary to meet customer and regulatory requirements.

  • Auditor should ask themselves:

    • Does the absence of the where appropriate, applicable, or necessary requirement put the entity at risk of not meeting requirements or reliability objectives.

    • Is determination of this entity different from similar entities

  • Entity should be able to provide some justification for “How” they came to a conclusion or made a decision related to subjective requirements.


Training records

Training Records

  • These have been pretty good

    • Generally sufficient

    • Generally appropriate

  • A few difficulties we see:

    • Validation that course listed actually contains required Emergency Training or Simulation

    • Sometimes get large volumes of copies of individual course certificates.

    • Training records do not always cover all training performed


Your company s roadmap

Your Company’s Roadmap

Quality Evidence begins with a document stating how your company will comply with a standard.

  • Includes the standards and requirements

  • Who or which department is responsible

  • What specific measurements need to be monitored

    Document for BAL-002-MX-0 Might include:

  • The standard requirements clearly stated.

  • Engineering or EMS groups may be responsible to ensure the calculations being performed by the EMS system are correct.

  • Operations may have direction to maintain ACE within specific limits


Your company s roadmap1

Your Company’s Roadmap

The document should describe what type of controls are in place to ensure compliance is achieved.

Document for BAL-002-MX-0 Might include

  • Guidance on how changes to EMS is approved and controlled such that inadvertent changes are not made which would result in inaccurate or lost data.


Your company s roadmap2

Your Company’s Roadmap

The document should identify what records or methods would be used to prove that compliance was achieved.

Document for BAL-002-MX-0 Might include:

  • A periodic review of data quality

  • A periodic review of Compliance Factors (CF)

  • A periodic review of CPS2 10 minute violations

  • Documentation of these reviews


Electronic documentation

Electronic Documentation

  • Key issues

    • Consistency of format

    • Reliable media

    • Search capable

    • Locked Documents (unable to open)


Pending standards

Pending Standards

EOP-003-MX-0 Load Shedding Plans

PER-001-MX-0 Operating Personnel Responsibility and Authority

PER-003-MX-0 Operating Personnel Credentials

  • All of the above are applicable to BA and TOP

    TOP-004-MX-0 Transmission Operations

  • Applicable to TOP


Questions

Questions

Phil O’Donnell

Manager, Operations and Planning Audit Team

Western Electricity Coordinating Council7600 NE 41st Street, Suite 160

Vancouver, WA 98662

360.567.4057

[email protected]

87


  • Login