ratc arpa e review meeting ratc communication system n.
Skip this Video
Download Presentation
RATC/ARPA-E Review meeting RATC Communication System

Loading in 2 Seconds...

play fullscreen
1 / 33

RATC/ARPA-E Review meeting RATC Communication System - PowerPoint PPT Presentation

  • Uploaded on

RATC/ARPA-E Review meeting RATC Communication System. Dr. Sami Ayyorgun – ACS Dr. Alex Sprintson – TAMU Dr. John Sucec – ACS . RATC communication system. Addresses RATC communication needs Support data flows among RATC components

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'RATC/ARPA-E Review meeting RATC Communication System' - billie

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
ratc arpa e review meeting ratc communication system

RATC/ARPA-E Review meeting RATC Communication System

Dr. Sami Ayyorgun – ACS

Dr. Alex Sprintson – TAMU

Dr. John Sucec – ACS

ratc communication system
RATC communication system
  • Addresses RATC communication needs
    • Support data flows among RATC components
    • Support RATC performance requirements, e.g. Quality of Service (QoS) constraints
    • Support cybersecurity of RATC operations
  • Leverages existing substation and WAN communications infrastructure
    • WAN
ratc communication needs
RATC communication needs
  • Intra-substation
  • substation – control center
ratc data flows2
RATC Data Flows

IEEE 1379



(ICCP or IEC 60870-6/TASE.2

progress since june 29 review
Progress since June 29 review
  • For each data flow
    • The source of data
    • The destination of data
    • Periodic/On Demand
    • Estimated size of the data
    • Delay requirements/ QoS constraints
current status and actions
Current status and actions
  • Establish connections to existing standards
current status and actions1
Current status and actions
  • Finalize the system and communication architecture of SCADA
    • Centralized approach
      • Most of the data processing is done at the central substation
    • Decentralized approach
      • Substations can perform certain operations
        • Measurements and Model Data Integration
        • Circuit breaker Operation Evaluation
actions until the next meeting
Actions until the next meeting
  • Finalize the data flows
  • Mapping to physical architecture
  • Implementation in typical utilities
  • Analysis of QoS requirements

TAMU (Sprintson)

ACS (Ayyorgun)

  • Identification of data flows
  • Identification of Quality of Service (QoS) constraints
  • Participate in protocol section and developing new protocols if needed
  • Analysis and evaluation of proposed solutions
  • Identification of RATC data and control flows.
  • Selection of protocols.
  • Identification of communication architectures.
  • Identification of communication performance requirements.
  • Identification of communication resource-adaptation needs.
  • Identification of performance and cybersecurity vulnerabilities, and potential mitigation approaches.
progress since june 29 review and ongoing work
Progress Since June 29 Review and Ongoing Work
  • Completed identification of candidate power systems operations (PSO) network architectures pertinent to RATC, including both physical and network topologies
    • Intra-substation and inter-substation communications
    • PSO WAN design
    • Intra-control center (CC) communications
  • Completed mapping of RATC network data and control flows to content sources/receivers and the network communication paths connecting them
  • Began specification of RATC substation communications requirements
    • RATC communications within the substation; communications between the SA RATC elements and CC RATC over the PSO WAN
  • Began mapping of network flow event sequences
  • Began evaluation of potential risks/issues/challenges including assessment of the following as they relate to the communications architecture, e.g.:
    • QoS-related performance goals, network congestion, network availability, cyber-security vulnerabilities
global view of ratc data flows and protocols
Global View of RATC Data Flows and Protocols


Intra-CC flows

Flows between substation and CC

Inter-substations flows

Intra-substation flows




summary of ratc network data and control flows
Summary of RATC Network Data and Control Flows
  • RATC modules will leverage the IED and model data maintained by existing utility substation data concentrator (SDC) and control center data concentrator (CCDC) systems but in some cases may need to acquire certain PSO data on its own.
  • Distributed RATC software processes also are expected to communicate with one another
    • RATC control signals computed by a CC RATC element will be disseminated to substation automation (SA) RATC elements situated at remote substation and power generation sites
    • Furthermore, local SA status and measurement data collected SA RATC elements will also be potentially backhauled to the CC RATC element
example ratc data flow data collection for circuit breaker operation evaluation
Example RATC Data Flow: Data Collection for Circuit Breaker Operation Evaluation
  • SA-RATC SW App triggers local circuit breaker operation evaluation (CBOE) data collection
  • SA-RATC CBOE SW App calls RATC Comms module for latest circuit breaker monitor (CBM) data
  • SA-RATCComms module:
    • Identifies SA LAN path and communication device adapter
    • Invokes SA-RATC Resource Adaptation to verify LAN conditions support request
    • Invokes RATC Cyber-Security to verify security policies permit request
  • SA-RATC Comms module completes CBM data transfer from CBM IED
  • SA-RATC CBOE SW App compresses / filters CBM data and calls RATC Comms module for transfer to CC-RATC
  • SA-RATC Comms module:
    • Identifies WAN path, CC-RATC comms port and DSCP marking
    • Invokes SA-RATC Resource Adaptation to verify WAN conditions support request
    • Invokes SA-RATC Cyber-Security to verify security policies permit request
  • SA-RATC Comms module completes compressed CBM data transfer to CC-RATC Comms module
  • CC-RATC CBOE updates CBM event records repository
internetworking ratc communications between control center and remote sites
Internetworking RATC Communications Between Control Center and Remote Sites
  • The key communication content pertinent to RATC between substations and the Control Center (CC) is the SCADA information employed by a utility’s energy management system (EMS)
  • RATC elements situated at the CC likely will share network resources with EMS elements
  • RATC communications between CC-RATC and SA-RATC elements will share the utility WAN resources with SCADA communications
  • Data flow: SA-RATC agents send periodic status and event-driven alerts to CC-RATC; CC-RATC sends topology control commands to SA-RATC agents
  • Risk: If a WAN path between CC and substation be congested, critical RATC event or command may be lost or unacceptably delayed.
  • Mitigation(s): DSCP values of RATC data packets should be marked according to priority/urgency and utility WAN should support DiffServ. RATC-specific communication control algorithms need to be deployed.
ratc communications security considerations
RATC Communications Security Considerations
  • Risks: Compromised communication assets (links and devices) could lead to wrong input data (e.g. load-flow data) and wrong output data (e.g. switching instructions) to be sent and received. This could lead to wide-area system instabilities.
  • Mitigation: RATC cyber-security considerations should be part of the RATC communication architecture design. So far, we have discussed RATC communication architecture from performance perspective.
    • Protocol AND application (RATC) specific, stateful cyber-security mechanisms need to be developed for RATC.
    • RATC security agents need to be installed at each substation and the CC.
    • Enterprise security solutions are not sufficient to protect various vulnerabilities (in particular for communications both intra and inter substations). This is due to performance restrictions (e.g. involving encryption).
    • One novel approach (that we will pursue) is to develop finite-state-machine models of proper behaviors of protocol (e.g. 61850) actions specific to RATC (e.g. control of RATC object models). These FSMs would be used as “sensors” to detect potential attacks and to coordinate mitigation strategies.
example to ratc security agent fsm sensor
Example to RATC Security Agent FSM Sensor
  • For communications and coordination among RATC computers in CCs, we would create ICCP associations. RATC head-end may need to execute certain SBO programs/devices in both the associated control centers and related substations. An adversary could inhibit these operations by staging various attacks. The following is a FSM sensor example to detect such attacks.
  • We will determine potential vulnerabilities specific to RATC and develop a representative set of FSM sensors.
  • Another novelty about these FSM sensors would be to include physical states (not just protocol states, over which RATC communications take place).
projected additional progress for jan 2013 quarterly review
Projected Additional Progress for Jan 2013 Quarterly Review
  • Complete identification of communication flows and architectures
    • Complete flow sequence diagrams for each flow class
  • Continue work on communications requirements
    • Confirm QoS requirements (data throughput, delay)
    • Confirm cyber-security requirements
    • Identify intra-CC requirements
    • Identify inter-substation requirements
  • Draft architecture of RATC communication modules for:
    • Network resource awareness/adaptation
    • Cyber-security
  • Continue RATC communications risk assessment and resource adaptation
    • Argue preliminary mitigation measures
illustrative substation physical topology station wide ethernet lan with iec61850 8
Illustrative Substation Physical Topology: Station-wide Ethernet LAN with IEC61850-8
  • The figure below shows the complete migration switched Ethernet technology
  • Topology affords fully decentralized peer-to-peer communications among substation elements
  • Furthermore, access via the substation gateway router potentially supports direct communications between the control center (CC) and individual substation bay sensor elements
  • Relevant RATC data: DFR, DPR, CBM and PMU event records
  • Relevant standards/protocols: IEC 61850 object model, IEEE C37.118, MMS, TCP/IP, Ethernet extensions (e.g. IEEE 802.1q)
  • Risks/Issues:
    • For sub-millisecond delivery of time-critical IED data, switched network should be Gigabit Ethernet with proper QoS mechanisms.
    • Ethernet link for RATC data concentrator could be a bottleneck due to collection of IED data and dissemination of such data with the RATC modules and the CC.
  • Mitigation(s): Network elements should support IEEE 802.1p QoS extensions for time-critical IEC 61850 objects. RATC specific communication control mechanisms need to be developed.
illustrative substation logical topology redundant double ring scn architecture
Illustrative Substation Logical Topology: Redundant double ring SCN architecture
  • Risk: Limited physical access to remote makes SCNs highly vulnerable failure events
  • Mitigation: Figure here shows a modified (i.e. reduced number of switches) redundant parallel double ring logical topology with the breakout of elements of substation bay 1 presented in detail.
  • This logical topology addresses the issue of network availability
    • Here, any two Ethernet switch elements can fail and still the core of Ethernet switches will still remain connected
    • All network hosts/IEDs are dual-homed to protect against link failure to edge devices
  • Also, of interest here is the dual-homed connectivity of individual bay elements to the redundant Ethernet bay switches
    • This protects against single link failure events within an individual substation bay.
  • Communication performance disturbance analysis needed for substation networking
inter substation ratc communications
Inter-Substation RATC Communications
  • The figure below shows an example of inter-substation connectivity based on Gigabit Ethernet virtual private LAN service (VLPS)
  • Relevant RATC data: Notification of fault events and batch IED (e.g. DPR, DFR, CBM, PMU) data
  • Relevant standards/protocols: IEC 61850 object model, C37.118, MMS, TCP/IP, IEEE 802.1q
  • Risks/Issue 1: Even with Gigabit Ethernet VPLS, sub-millisecond delivery of data between substations over the WAN is not likely feasible
  • Mitigation: SA decisions that demand responses on the order of milliseconds after IED event, likely need to be made based on data already available to the local RATC agent
  • Risks/Issue 2: When high-speed WAN connectivity is not available, e.g. due to failure of primary link) or WAN coverage limited to slow (56Kbps) links, RATC inter-substation throughput will be low
  • Mitigations: RATC agents should be “resource-aware” and filter (e.g. drop) or mark (e.g. DSCP value) according to the priority/urgent of the inter-substation data
iec 61850 object model interface with protocol stack
IEC 61850 Object Model Interface with Protocol Stack
  • One option (for data transfer) is to create RATC data object model in 61850. We would leverage 61850 mappings for such an object model. This would provide unity and interoperability for RATC to work in 61850 compliant communication environments.
  • IEC 61850 defines mappings between SA data models and underlying communication protocols:
    • Mapping of time-critical raw measurement data samples and Generic Object Oriented Substation Event (GOOSE) data to the Ethernet link layer
    • Mapping of client-server communications to the Manufacturing Message Specification (MMS) protocol which is an industry standard protocol that operates at the Application Layer in the TCP/IP protocol stack
substation comm transition to ethernet
Substation Comm. Transition to Ethernet
  • The transition to Ethernet-based substation communication network (SCN) architectures, that RATC would benefit, provides:
    • Switched Ethernet offers attractive scalability and ease of management/maintenance advantages over conventional wiring
    • The advancement of high-speed Ethernet technology helps with low-latency delivery needs of time-sensitive raw measurement data and GOOSE events
    • The deployment of IEC 61850-compliant intelligent electronic devices (IEDs) provides an interface to the TCP/IP stack (and Ethernet) that was not available in legacy devices.
  • For legacy systems, we could add protocol convertors for 61850 over Ethernet for RATC devices. However, this only helps with interoperability and unit, not necessarily with performance. For performance, communication resource adaptation for RATC in legacy environments play a critical role.
ratc control center communications
RATC Control Center Communications
  • The figure here shows the key functional areas of the CC EMS plus its interactions with the control center business management system (BMS) and remote site operations
  • Data Flow: As discussed previously, CC-RATC will share CC EMS network resources to obtain SCADA/RTU information, accessing the Control Center Data Concentrator (CCDC); Furthermore, the CC-RATC processes will also access Model data (static, dynamic) including model data (line, generation, load) and current state updates
  • Risks: Unconstrained CC-RATC over-the-EMS-network-access to the voluminous quantity of data maintained at CC could potentially disturb CC network performance.
  • Mitigation(s): RATC intra-CC communications should incorporate network resource-awareness procedures (TBD) to intelligently access data based on the available CC network capacity.
ratc wan design considerations remote site connectivity using utility owned infrastructure
RATC WAN Design Considerations: Remote site connectivity using utility-owned infrastructure
  • The figure here shows utility-owned non-backbone WAN connectivity to remote utility sites – e.g. TVA system.
  • Depending on site network reliability and performance requirements, a mix of link technologies may be deployed to support disparate site internetworking needs
  • Key advantages of the utility-owned infrastructure include the ability to reach remote sites not serviced by public networks and improved cyber-security
  • Of particular importance to RATC communications is the ability for the utility network operator to manage how traffic is serviced with DiffServ according to a packet’s priority/urgency as indicated by it DSCP field
  • Communication performance disturbance analysis is needed for WAN as well.
ratc wan design considerations wan connectivity with non utility infrastructure
RATC WAN Design Considerations: WAN connectivity with non-utility infrastructure
  • Risk: Using non-utility-owned WAN resources means less control over network flow QoS
  • Mitigation: RATC WAN flows should be deployed in close coordination with other utility flows so as to use WAN resources in a manner that protects urgent/priority data
relevant power systems operations data items
Relevant Power Systems Operations Data Items
  • The RATC algorithm modules use various types of intelligent electronic device (IED) measurement data and utility energy management system (EMS) model data
  • Estimates of data item polling rates and data item sizes are based (in part) on discussions with TVA
  • The data update rates required by the RATC algorithm are currently under study by the RATC team