Summary of Findings and Recommendations

1. Summary of Findings and Recommendations May 21, 2002

2. In the handout • A more detailed report on findings and recommendations dated May 13th • An appendix containing an interim findings discussion document dated May 2nd

3. Today’s Discussion Agenda • Summary of Findings • Summary of Recommendations • Questions

4. Summary of Findings

5. Summary of Root Causes of Failures Probable • Start up pains for all market participants • New transaction sets • New processes • New software • Many participants with many systems (new and legacy) • Tight time lines • New people in many jobs • Many “cooks in the kitchen” • Almost everybody is in some way learning how to do this for the first time • Limitations in recognition and “graceful handling” of unexpected conditions in ERCOT systems exacerbated by: • Extraordinary transaction volumes • Transactions out of sequence or missing from sequence as defined by protocol • Limited ability to generate/communicate rejections to market • ERCOT systems as defined and implemented are are not built to gracefully handle syntactically correct, but logically invalid TX SET transactions

6. Summary of Root Causes of Failures • Occurrence of changing or evolving requirements definition during the later stages of system design and development. • Limitations inherent in the approach to testing • Testing did not adequately cover • Unexpected/aberrant data • Out of sequence transactions • Out of protocol delays • Legacy systems dependent on transaction flows • Abbreviated integration testing across all affected market participants • Inadequate end-to-end automated transaction flow management, monitoring and logging • “Leakage” • Not adequate as currently implemented • There should be zero “lost” transactions • A given transaction’s history and current status should be available at any point in time

7. Summary of Root Causes of Failures • Side effects of “Safety Net” workarounds • Out of sequence transactions hitting ERCOT systems • ERCOT and market participant data out of sync resulting in future settlement problems • Some parts of underlying IT application architecture appear to limit throughput resulting in delays in transaction processing

8. Other Key Issues • Performance measurement in key transaction handling • Need to more completely define: • What is necessary/expected of all parties? • What are the related key measures and performance indicators? • Not just failed transactions, but also throughput • How we will measure them? • How we will ensure continuous improvement? • Focus should be on improvement, not punishment • Service Level Agreements, driven by the “real life” needs of the market must be solidified • Role of ERCOT • Market needs Initiator/Coordinator/Driver for working key market issues • Through key decision-making bodies (BOD, TAC, RMS, WMS, PRS) • With all Market Participants (TDSPs, CR, QSEs etc) • Acting as technical arbitrator (TDSP – CR) • Facilitator of thought leadership • Maintaining and actively championing a market-wide perspective • Taking an holistic approach • Roles evolving for ERCOT will likely require targeted increases in resources and in some cases different/stronger skill sets

9. Other Key Issues • Volume of current recovery efforts is limiting resources available for efforts to achieve prevention • ERCOT and especially ERCOT IT needs to continually look for ways to reduce the burden on the market participants in utilizing ERCOT services especially where doing so could improve the overall reliability of the transaction flow and the cost is acceptable • With the market participants, ERCOT needs to define the acceptable risk profile in the areas of high availability/disaster recovery • Must agree to level of risk all parties will assume • Must agree on cost of agreed to profile • Need to bring formal closure to the Accenture / System Acceptance / Warranty related questions

10. What ERCOT does primarily…….. • Safely, efficiently, and effectively manage the flow of power across the electrical grid in covered areas within Texas • Manage the fair, expedient, efficient and reliable exchange of information needed to support all market transactions among the market participants, as well as provide central registration services This is a business in its own right.

11. Summary of Recommendations

12. Areas of Recommendations • Communications across the Market • Market Transaction Processing Pipeline • Monitoring • Performance • Scorecard • Workarounds • Functionality • Other ERCOT IT and related issues • Other ERCOT issues • ERCOT’s role in the Market

13. Communications across the Market • Improved market wide communication is needed to improve both problem prevention and problem recovery, as well as, consistent understanding of current status and performance of key transaction processing activities. • This includes: • Unplanned events • Planned events • Technical issues • Business issues • Transaction status • Performance of key transaction processing services

14. Areas of Recommendations • Market Transaction Processing Pipeline • Monitoring • Performance • Scorecard • Workarounds • Functionality

15. Retail Market Transaction Processing Pipeline Definition Market Participants

16. ERCOT’s visibility into the Retail Market Transaction Processing Pipeline Market Participants Visibility is limited by current capabilities of the underlying transaction processing software as implemented, especially at the interface point between major software modules.

17. ERCOT’s visibility into the Retail Market Transaction Processing Pipeline Market Participants Monitoring should provide Component Health (Up/Down) Performance & Throughput Exceptions & Events Transaction Status

18. ERCOT’s visibility into the Retail Market Transaction Processing Pipeline Market Participants Monitoring Component Health (Up/Down) Performance & Throughput Exceptions & Events Transaction Status

19. Monitoring of Market Transaction Processing Pipeline Ensure the monitoring of key points and components within ERCOT’s transaction processing environment to be used in managing: • Availability • Performance/Throughput • Exceptions • Events

20. Monitoring of Market Transaction Processing Pipeline (cont.) • “Beginning to End” monitoring of all key components and interface points within the transaction processing pipeline must be put in place • Increased market-wide visibility • ETS project • Retail market portal • A “dashboard” must be defined and implemented that tells us the current status, health and performance of the transaction processing pipeline • As dashboard alerts occur, a universally understood process must be followed for diagnosing and addressing the alert as well as reviewing for root cause determination.

21. Performance of Market Transaction Processing Pipeline Need to improve the performance of ERCOT’s transaction processing and central registration services as defined by: • Availability • Performance/Throughput • Compliance with established protocols • Efficiency of operations • Number of exceptions and the effective handling of those exceptions

22. Performance of Market Transaction Processing Pipeline (cont.) • Solidify the FTP replacement project currently underway to improve visibility and reliability of that portion of the pipeline between ERCOT and the market participants • Continue efforts to evaluate all existing hardware and software components to determine potential for upgrading to improve performance • Review all key processing software components beginning with Siebel and PaperFree, to determine opportunities to establish multi-threading processing capability

23. Market Transaction Processing Scorecard • Develop and publish a Market Services Performance Reporting Scorecard • Define metrics and measures for successful Market Services operations. Establish appropriate Service Level Agreements (SLAs) • Begins with - What is success in the market? • Includes - What is success in the systems and processes supporting the market? • Define format, content, audience, and ongoing publishing requirements for scorecard • Collect relevant current volumes, success & error rates, and performance metrics • Measure against success criteria and publish scorecard via the WWW

24. Retail Market Transaction Processing Pipeline now has Workarounds outside of the original design Market Participants Workarounds

25. Market Transaction Processing Workarounds • Fully understand the current workarounds and develop missing components to mitigate the adverse side-effects • Ensure workarounds are either institutionalized or eventually eliminated by new transaction processing pipeline improvements • Identify additional naturally occurring transactions that appear out of sequence to the protocol specifications, and develop workarounds as well as defining permanent solutions to be implemented in subsequent software releases

26. Market Transaction Processing Functionality • Develop and implement plan to “harden” current transaction processing pipeline within ERCOT by: • Improving key error detection, notification and handling processes including EDI rejection detection, generation, and processing • Strengthening data validation in ERCOT systems specifically in Lodestar and Siebel • Ensuring the ability to reliably deliver, receive, trace and provide status of all transactions in and out of ERCOT • Reducing the constraints and limitations placed on market participants by ERCOT regarding the exchange of data

27. Market Transaction Processing Functionality (cont.) • Future enhancements to the transaction processing pipeline must be properly sequenced and integrated • TX SET 1.5 should be reviewed with the intent of first hardening the current transaction pipeline to establish a solid foundation upon which to add new functionality such as “stacked moves”

28. Areas of Recommendations • Other ERCOT IT and related issues

29. Other ERCOT IT and related issues • An increase in the capacity of ERCOT IT to resolve problems and build new functionality into the transaction processing pipeline is needed • Remove obstacles to getting the most important work done and done well • Additional IT resources within ERCOT are likely to be required • One integrated list of all needed IT work/requests with a common agreed-to basis for setting priorities must be established and managed against • A more complete and comprehensive approach to defining, estimating, costing and ensuring benefits are obtained in IT investments should be put in place • ERCOT must develop, own and manage testing methodologies, “test cases” and the execution of testing of changes to the transaction processing pipeline. The scope of the ongoing testing must be materially expanded

30. Areas of Recommendations • Other ERCOT issues

31. Other ERCOT related issues • The capacity of ERCOT’s Retail Market Services support organization must be immediately expanded • Increased resources (ideally with contractors, probably Accenture) should be put in place within the existing “fix it” team to increase rate of problem resolution and prepare for summer volume • Improve the utilization of ERCOT Retail Market Services support resources by: • Leveraging the recently revised Tiger Team processes for prioritizing, researching, and resolving market issues. • Continuing the Tiger Team problem research and problem resolution efforts at current staffing levels. • Begin moving Tiger Team responsibilities into Market Services as their capacity increases • Overall improvement in problem management across market is needed

32. Areas of Recommendations • ERCOT’s role in the Market

33. ERCOT’s Role in the Market • ERCOT should have an expanded role in the market defined by: • Assuming primary responsibility for ensuring the successful execution of standardized operations within the market • Providing the primary leadership in diagnosing, resolving and preventing transaction processing problems in the marketplace • Being the primary driver and facilitator of recognized market driven changes across the relevant market participants, committees, subcommittees, agencies, vendors and ERCOT Board of Directors • Owning, driving and being the authoritative “keeper” of core market processes and their ongoing evolution

34. Next Steps for ERCOT and the Market Participants • Confirm priorities as presented • Launch near term efforts immediately • Develop a formal plan including more detail estimates of what is required in terms of: • Time • Money • Resources • Determine feasibility from a financial and personnel perspective • Determine staffing implications and establish ownership/accountability • Launch longer term investment efforts