Faculty: Dave Bakken, Carl Hauser, Anjan Bose

GridStatMiddleware for More Extensible andResilient Status Disseminationfor the Electric Power Grid Faculty: Dave Bakken, Carl Hauser, Anjan Bose Students: H. Gjermundrød, I. Dionysiou, R. Johnston P. Jiang, S. Sheshadi, K. Swenson School of Electrical Engineering and Computer Science Washington State University Pullman, Washington USA http://gridstat.eecs.wsu.edu November, 2003

Overview of Presentation • Background and Motivation • GridStat Architecture • Gridstat Implementation and Deployment Issues Dave Bakken

Power Grid Today Overview • 3 fundamental roles • Historically one vert-integr. utility • IT/control based on this fixed hierarchy Hierarchy • Substation • Control Area/utility • Grid Generation TransmissionSubstation Subtransmission Transmission Substations Distribution Distribution Substations Customers (Create load) Figure credit: NSTAC Residential Commercial Industrial Dave Bakken

Protection and Control Today are Local • Remedial Action Schemes (RAS): hardwired remote link to trigger a protective relay • Otherwise almost exclusively local monitoring (status) & local control • Power dynamics are grid wide, and anomalies can affect a wide geographic area Dave Bakken

The Changing Landscape • Higher demand for power transmission – miles x megawatts • More power and longer distances with little new transmission capacity • Installed transmission capacity constrained by minimum of • Thermal limit • Stability limit • More participants whose actions affect grid stability • Technology in recent years is adding • Many more devices “intelligent” devices • Much more heterogeneity • Lack of central authority • Terrorist concerns • Labor disputes, Environmental Liberation Front, everybody is an insider (or can be with a little effort), … Dave Bakken

Power Companies Are More Interrelated • Traditional generation + transmission + distribution is no longer inside a single utility • Utilities can be affected by many things they cannot sense/detect/measure with today’s communication infrastructure • Interactions between power dynamics and grid communicaiton dynamics completely unknown Dave Bakken

ISO and Grid Security • Independent System Operator (ISO): layer above the control area layer added in the last few years • A small number of ISOs for bigger grids • Real-time balancing of supply and demand • ISO is responsible for grid security • Means no actions being considered, or any probable contingency, can lead to a blackout or brownout • Roughly translates to what computer scientists would consider stability and reliability • Grid security is an on-line, real-time activity • ISO monitors status from all control areas • Receives status info from control areas and substations in its jurisdiction • ISO’s functionality previously performed by the vertically- integrated utilities • Now too much power flowing across and around them Dave Bakken

Status Information & the Power Grid • Changing requirements • More general topology and connectivity including multicast • Existing hardwired, hierarchical structure does not suffice • Status items may be needed at multiple locations • New services require more quantity, timeliness, … • Improved real-time controls to push stability limits nearer thermal limits • Situation awareness: phone calls not adequate! • 4-second SCADA cycle moving to ½ to 4 times per 60 Hz cycle • Opportunities for new kinds of automatic real-time control • Local control of devices based on remotely sensed status • Closed loop controls with relatively long delays • Stochastic control Dave Bakken

GridStat in a Nutshell • SCADA is a distributed computing problem • Convey status data in a reliable, timely and secure manner (QoS) • Above the network layer • Implementing these QoS properties requires considerable sophistication • Exploit application-level semantics and QoS requirements • Want a re-usable architecture • Compare to Web:TCP as SDM:network layer • GridStat: status dissemination middleware tailored for the power grid • Common service platform for disseminating power grid status information within and between power utilities, marketers, etc. • Timely, robust, and secure delivery to multiple participants • Collaborative project with CS and EE at WSU, others • Also applicable to status dissemination needs of other infrastructures: transportation, water, gas, … Dave Bakken

GridStat is Publish-Subscribe Middleware • Publish-subscribe architecture • Publish: periodically announce status values • Subscribe: periodically receive status values • Simple, CORBA-compliant APIs for both publishers and subscribers, management/control infrastructure, etc. • Subscribers have transparent cache of latest status value • Network of internal servers managed for QoS • Optimized for semantics of status items • Not just arbitrary event delivery like generic publish-subscribe Subscriber #1 Subscriber #2 … Publisher Subscriber #N Dave Bakken

What is Middleware? • Middleware == A layer of software above the operating system but below the application program that provides common programming abstractions for distributed systems • Middleware exists to help manage the complexity and heterogeneity inherent in distributed systems • Middleware provides higher-level building blocks for programmers than the OS provides • Makes code more portable • Makes programmers more productive • Final product is of higher quality • Analogy — MW:sockets ≈ HOL:assembler Dave Bakken

GridStat Middleware (MW) in Context Host 1 Host 2 Distributed Application Distributed Application Client Client MW Router MW Router MW Router Middleware API Middle- ware Middle- ware Middleware API OS Oper. System API Oper. System API OS Comm. CPU Storage Storage CPU Comm. MW QoS Management control control Wide-Area Network Dave Bakken

Fundamental GridStat Research Issues • Status dissemination middleware is a new specialization of publish-subscribe MW • Recognize specialized requirements of status dissemination • Take advantage of status semantics in order to meet those requirements • What are the APIs? What promises are made by the middleware to the application regarding functionality and performance? • What architecture can deliver on these promises? • How can we validate the correctness, timeliness, quality, etc, of a concrete embodiment of the architecture (the framework)? • What are the trust issues between grid participants. What policies are required? How can they be implemented? • How can the architecture be made economically scalable and manageable • Goal of GridStat resarch is to begin answering some of these questions and embody those answers in the GridStat middleware framework Dave Bakken

Overview of Presentation • Preliminary Information • GridStat Architecture • Gridstat Implementation and Deployment Issues Dave Bakken

Publication and Subscription 3.75 3.79 3.76 • Status variable – located at a publishers • Periodic sequence of time-stamped values or • Sporadic sequence of time-stamped alerts • Types supported (initial, illustrative set): • Basic values: scalars of type bool or int or float • Derived values: moving average, change rate, moving average of change rate, max or min over an interval • Derived values are first-class: subscribe to them just as to a basic value • Subscription – requested by a subscriber • A promise (by the MW) to deliver values from a particular status variable at a given rate within the requested delay (timeliness) • Subscriber’s rate must not exceed publisher’s rate • Delay is constrained by the network time3 time2 time1 27.93 27.56 24.33 time3 time2 time1 alert time1 Dave Bakken

Basic GridStat Functionality QoS Requirements QoS Requirements Control Generator Area Controller Load Following ISO Department of Homeland Security Grid Area Controller GridStat Management Wide Area Computer Network … … Subscribers Publishers Dave Bakken

Overview of GridStat’s Architecture … QoS Requirements … … R R R R QoS Requirements R Control R Control R R Pub1 Sub1 … … PubN SubN QoS Broker QoS Broker QoS Broker GridStat delivers status events from publishers to subcribers Dave Bakken

Detailed Architecture Key: Publisher 1 Publisher 2 Publisher 3 Dave Bakken

Route Allocation to Subscriber1 Key: Subscriber 1 Publisher 1 Publisher 2 Publisher 3 Dave Bakken

Route Allocation to Subscriber2 Key: Subscriber 1 Publisher 1 Publisher 2 Publisher 3 Subscriber 2 Note: Sub2 may have a different rate or latency than Sub1 Dave Bakken

GridStat & UCA v2 & SCADA Co-Existence Key GridStat Status Data (phone calls today) Homeland Defense GridStat QoS Control (later status aggregation) ISO ISO Other Critical Infrastructures Utility #1 Control Center Utility #2 Control Center Note: UCA v2 has no wide-area network mgmt, though it discusses exploiting “future communcations services” UCAv2 Substation UCAv2 Substation SCADA Substation SCADA Substation Dave Bakken

Programming Model: GridStat Subscriber Caches … Status Router Status1 Status1 Subscriber1 Status2 … Status2 … cache1 Status Router Status2 Subscriber2 … Status3 … Status3 cache2 Dave Bakken

Programming Model: Condensation Functions Status Router • If desired derivation is not built in, condensation functions allow applications to define new derived status variables • Sometimes subscribers just read a large set of status items once to calculate a derived variable • Supported by allowing user-defined condensation functions to be loaded in status routers Status1 … StatusJ Condense StatusN Dave Bakken

Overview of Presentation • Preliminary Information • GridStat Architecture • Gridstat Implementation and Deployment Issues Dave Bakken

Pragmatic Deployment Feasibility Notes • GridStat not completely deployable using best-effort internet technology • But much of the traffic might be best effort, especially with bandwidth reservation • Most likely deployment path: grid-wide intranet • Washington Post: “In a book-length Electricity Infrastructure Security Assessment, the industry concluded on Jan. 7 that "it may not be possible to provide sufficient security when using the Internet for power system control." Power companies, it said, will probably have to build a parallel private network for themselves.” (emphasis mine) • Not only would security be lacking, but predictable timeliness and resilience too!!!!! The Internet is arguably as complex as the grid! GridStat Team Opinion • Some portions likely co-located with tel-cos and national ISPs’ facilities; others built using private facilities of electric utilities • In addition to short-term application, GridStat should be viewed as a platform for exploring what services to provide in status dissemination middleware Dave Bakken

GridStat Capabilities Today • Static routing of status variables to meet subscriber’s timeliness and redundancy requirements • Recovery from data link and management link failure • Hierarchical QoS brokers • Graphic visualization of status items (strip charts etc) and of the internals of leaf QoS brokers (queues, etc) • Note: GridStat could deliver remote control commands in addition to status data • Just another kind of data to deliver…. Dave Bakken

GridStat Prototype • Finished: 2nd-Generation Distributed Prototype with • Hierarchy of QoS Managers performing the allocations • Publisher delivery rate & redundancy QoS requirements satisfied • Optional exception callback to subscriber of QoS violated Dave Bakken

Future GridStat Capabilities (Funding Pending) • Fault Tolerance at many more levels • Broader QoS Routing with runtime feedback • Trust management system to allow secure runtime subscriptions • Pre-allocated subscription “packages” for rapid deployment in contingencies • Validation framework (SW quality, QoS delivery) • Hardware support • Modeling and control theory for communication dynamic and power dynamics interacting Dave Bakken

Ongoing & Future Research Issues • Investigating a range of optimizations • Periodic status items only delivered if enough change • percentage • fixed delta • Throttle back lower priority status flows when overload, attack, accidents, etc. using subscription info (max timeliness, min redundancy) • Subscription aggregation of different kinds of flows and sub-flows • Resilience • Subscriber cache extrapolation • Adaptive path management • Push the data path into hardware or embedded processors • 10% of code perhaps • Status routers, HW registers for publishers and subscribers Dave Bakken

Collaborators, Funding, and Colleagues • Faculty: David E. Bakken, Carl Hauser,Anjan Bose, • Students: Ioanna Dionysiou, Kjell “Harald” Gjermundrød, Thomas Evje, Ryan Johnston, Supreeth Sheshadri, Ping Jiang • Funding: • US Dept. of Commerce, National Institute of Standards and Technology (NIST), Critical Infrastructure Protection Program, Grant #60NANB1D0116 (Dr. Tim Grance, PM) • The National Science Foundation, Grant CCR-0326006 (Dr. Helen Gill, PM) • Pending: DHS, soon NSF ITR (Feb 03) • Collaborators: • Prof. Kevin Tomsovic, WSU, realtime grid control with varying feedback loops and varying time horizons • Prof. Sandip Roy, WSU, control theory with stochastic delays • Prof. Deborah Frincke, U of Idaho, security and trust • Other WSU professors with interests in temporal queries, hardware implementation, graph theory, software engineering • CMU/CERT: Easel Simulation System and GridStat; IT modeling for power grids, … Dave Bakken

GridStat & Avista • Now working with Avista Utilities to experiment with distribution status data dissemination • Utility for WSU’s area, with presence in 5 western states • Technology demonstration deployment underway • Avista has donated $2.4M in dark fiber around • Spokane area • Pullman (WSU) • Moscow, ID (U. Idaho; 8 miles from WSU) to support GridStat and similar research and distributed evaluation at WSU, U. Idaho • Funds for “access points” and Avista engineering labor also provided Dave Bakken

Related Work • Computer Science (networking, distributed computing): • PASS (BBN/Gatech ICDCS ’99 Zinky/O’Brien/Bakken/…) • Sienna (U. Colorado): content-based publish-subscribe • InfoPipes (GaTech): fresh delivery of status info • SpinGlass/Astrolabe (Cornell): scaleable multicast • Electrical Engineering (power) • A few research papers pointing out the wide-area communication deficiencies • UCA version 2: nice “wrapping” of substation devices, but no QoS management across WANs. Dave Bakken

Conclusions • Existing power grid SCADA/DCS infrastructure is not adequate • Deregulation and restructuring • Efficient use of transmission resources • GridStat: status dissemination middleware tailored for the power grid • Publish-subscribe architecture with simple, CORBA-compliant APIs for both publishers and subscribers • Subscribers have transparent cache of latest status value • Network of internal servers managed for QoS • Timeliness • Redundancy • Security Dave Bakken

Questions? Dave Bakken

Background Slides • Power Grid 101 • Washington Post quotations on power grid cyber attacks • Middleware 101 • More GridStat Details Dave Bakken

Context “The ultimate challenge in creating the power delivery system of the 21st century is in the development of a communications infrastructure that allows for universal connectivity.” “In order to create this new power delivery system, what is needed is a national electricity-communications superhighway that links generation, transmission, substations, consumers, and distribution and delivery controllers.” Clark Gellings, EPRI Vice President for Power Delivery and Markets, in “Smart Power Delivery ― A vision for the Future,” EPRI Journal Online, Electric Power Research Institute, June 9, 2003 http://www.epri.com/journal/details.asp?doctype=features&id=618. GridStat is researching, implementing, and evaluating this “national electricity-communications superhighway”, and not just for the power grid but for other critical infrastructures as well. Dave Bakken

Power Grid Today • Three fundamental roles in the power grid: • Generation • Transmission • Distribution • Traditionally owned by a single, vertically-integrated company • Based largely on geography • Hierarchical infrastructure • Communications network is • Hardwired • Dedicated • Slow • Everything is hard-coded based on this fixed hierarchy • Application programs • Status information • Control decisions Dave Bakken

Components of the Power Grid • Generator: generates power, based on requirements given it • Substation: point of monitoring and control in the grid • Can service many generators, and/or other functions • Distribution point to customers • Voltage boosting • Control functions • Generally only services one fundamental role • Always involved in control based on status of a lot of devices • Control area: a set of substations • Geographic area ranging from a county to a few states • Services all three fundamental roles • Roughly corresponds to one or a few utility companies (most 1:1) • Collects status info from all substations for control decisions • Grid: a set of control areas which are synchronously controlled • AKA “regional reliability council” or “region” Dave Bakken

Grids in Canada and the US Dave Bakken

ISO and Grid Security • Independent System Operator (ISO): new layer above the control area layer currently being added • A small number of ISOs for bigger grids • ISO is responsible for grid security • Means no actions being considered, or any probable contingency, can lead to a blackout or brownout • Roughly translates to what computer scientists would consider stability and reliability • Grid security is an online, real-time activity • ISO monitors status from all control areas • Receives all status info from any control area or substation in its jurisdiction • ISO’s functionality used to be performed by the vertically- integrated utilities • Now too much power flowing across them or around them Dave Bakken

Pragmatic GridStat Deployment Feasibility Notes • Allocation algorithms & frequency of subscriptions • In practice nearly all are likely to be pre-allocated and static • Number of new subscriptions (allocation algorithm runs) per hour small • Could be batched for offline (weekend/night) computation unless critical • Even brute-force solutions to NP-hard problems may be practical in many cases Dave Bakken

Background Slides Washington Post quotations on cyber attacks “Cyber-Attacks by El Qaeda Feared” (27 Jun 02, A01)) • Note: emphasis mine and [comments mine] in all cases…. • “The event I fear most is a physical attack in conjunction with a successful cyber-attack on the responders' 911 system or on the power grid,” Ronald Dick, director of the FBI's National Infrastructure Protection Center Dave Bakken

Other Related Quotations from Post Article • The devices are called distributed control systems, or DCS, and supervisory control and data acquisition, or SCADA, systems. …. What is new and dangerous is that most of these devices are now being connected to the Internet -- some of them, according to classified “Red Team” intrusion exercises, in ways that their owners do not suspect. Because the digital controls were not designed with public access in mind, they typically lack even rudimentary security, having fewer safeguards than the purchase of flowers online. Much of the technical information required to penetrate these systems is widely discussed in the public forums of the affected industries, and specialists said the security flaws are well known to potential attackers. Dave Bakken

Post Quotations (cont.) • Digital controls are so pervasive, he said, that terrorists might use them to cause damage on a scale that otherwise would “not be available except through a very systematic and comprehensive physical attack.” [He is Director John Tritak of the Commerce Department's Critical Infrastructure Assurance Office] • To destroy a dam physically would require “tons of explosives,” Assistant Attorney General Michael Chertoff said a year ago. To breach it from cyberspace is not out of the question. In 1998, a 12-year-old hacker, exploring on a lark, broke into the computer system that runs Arizona's Roosevelt Dam. He did not know or care, but federal authorities said he had complete command of the SCADA system controlling the dam's massive floodgates. Dave Bakken

Post Quotations (cont.) • Massoud Amin, a mathematician directing new security efforts in the industry, described the North American power grid as “the most complex machine ever built.” At an April 2 conference hosted by the Commerce Department, participants said, government and industry scientists agreed that they have no idea how the grid would respond to a cyber-attack. What they do know is that "Red Teams" of mock intruders from the Energy Department's four national laboratories have devised what one government document listed as "eight scenarios for SCADA attack on an electrical power grid" -- and all of them work. Eighteen such exercises have been conducted to date against large regional utilities, and Richard A. Clarke, Bush's cyber-security adviser, said " the intruders “have always, always succeeded.” Dave Bakken

Why Middleware? • Middleware == “A layer of software above the operating system but below the application program that provides a common programming abstraction across a distributed system” • Middleware exists to help manage the complexity and heterogeneity inherent in distributed systems • Middleware provides higher-level building blocks (“abstractions”) for programmers than the OS provides • Can make code much more portable • Can make them much more productive • Can make the resulting code have fewer errors • Analogy — MW:sockets ≈ HOL:assembler • Middleware sometimes is informally called “plumbing” • Connects parts of a distributed application with “data pipes” and passes data between them Dave Bakken

Middleware in Context Comm. Comm. Processing Processing Storage Storage Host 1 Host 2 Distributed Application Distributed Application Client Server Middleware API Middleware Middleware Middleware API Operating System API OS OS Operating System API Network Dave Bakken

Middleware Benefit: Masking Heterogeneity • Middleware’s programming building blocks mask heterogeneity • Makes programmer’s life much easier!! • Kinds of heterogeneity masked by middleware (MW) frameworks • All MW masks heterogeneity in network technology • All MW masks heterogeneity in host CPU • Almost all MW masks heterogeneity in operating system (or family thereof) • Notable exception: Microsoft middleware (de facto; not de jure or de fiat) • Almost all MW masks heterogeneity in programming language • Noteable exception: Java RMI • Some MW masks heterogeneity in vendor implementations • CORBA best here Dave Bakken

Faculty: Dave Bakken, Carl Hauser, Anjan Bose