The BAA Terminal 5 Project – The reality of XML, ESB and Web services

1. The BAA Terminal 5 Project – The reality of XML, ESB and Web services

2. The reality of serving 35 million people per year • 37 million man hours to build T5 • 6.5 million cubic metres of earth works • 15,000 cubic metres of concrete per week • 16 major projects, 100 sub-projects • Sub projects cost between £30M and £150M • 60,000 people involved in the build • The IT infrastructure must operate entirely new level of speed, efficiency and availability (they land planes!)

3. The IT landscape – must be integrated • 6000 display systems, 400 COTS apps,197 line of business apps, 35 operationalIT platforms, over 1000 servers • One hour server failure has Europe-wide impact on flights, more than one hour has global impact • There have been two failures of over 6 hours in the last 8 months – these even caused several mile tailbacks on major surrounding roads • £250M fine for latedelivery of T5 !!!

4. Why not Web services alone? If we asked you to solve this problem with WS-*, SOAP and XML – what would happen?

5. How would we get there? Today’s architecture rigid, costly and difficult to operate • Proprietary technologies and skill sets • Multiple communication infrastructures • High cost of license, consulting and operation • Lots of turf control and organizational issues ORDERENTRY ERP CRM PARTNER SYSTEMS FINANCE

6. In walks MOM – and Web services PACKAGED APPLICATION & LEGACY SYSTEMS J2EE™ APPLICATION .NET™APPLICATION XML XML PARTNER SYSTEM WEBSERVICE Standard Interfaces are Major Step Forward • Hiding implementation details enables reuse • XML-based data easily exchanged • Designed for remote access, across heterogeneous platforms • Can be easily passed over HTTP(S), JMS, CORBA, Sockets, MQ, RV and almost any other messaging layer TCP/IP WEB SERVICESINTERFACE

7. Web Services PACKAGED APPLICATION & LEGACY SYSTEMS J2EE™ APPLICATION .NET™APPLICATION PARTNER SYSTEM WEBSERVICE But Have We Solved The Whole Problem? • Is it reliable, scalable and secure? • How do you change business processes? • How do you manage and monitor distributed services? TCP/IP WEB SERVICESINTERFACE Web services are interoperable communications stacks and don’t offer routing, service deployment, management, format transformation, guaranteed delivery, etc. You are building standards based spaghetti !

8. Why not Web Services alone? WS still lacks federated enterprise features • WS-Reliable Messaging creates reliable point-to-point connections • But still 100’s or 1,000’s of them – where’s the manageability? • How do you configure reliability to suit your needs? • WS-Security creates security “Swiss cheese”… • Each secure Web service needs to authenticate incoming messages • All accessing corporate security server? At the same time? • Creates 100’s of security holes? Can you run your Web services in the DMZ? • Many customers we work with simply do not allow external Web services

9. Where is the Strategic Inflection Point? When the balance of forces shifts from the old structure, from the old ways of doing business and the old ways of competing, to the new. Before the strategic inflection point, the industry simply was more like the old. After it, it is more like the new. It is a point where the curve has subtly but profoundly changed, never to change back again. - Andy Grove, Only the Paranoid Survive, 1996 By 2008, Gartner predicts that SOA will be a prevailing software-engineering practice, ending the 40-year domination of monolithic software architecture.

10. So, back to BAA…

11. BAA – The Strategy “… our strategy is to minimize the interdependencies between products, using open standards to increase operational flexibility and make sure that applications are responsive to change. Therefore a Service Oriented Architecture approach is inevitable. Our first challenge was to find a platform that would work well in our very demanding environment, and could orchestrate the services that will drive T5 operations. Sonic Enterprise Service Bus is a very natural fit." Nick Gains Head of IT BAA

12. Why did BAA choose SOA? For years, change velocity has outstripped IT capacity • New technology, regulations, re-organizations, and market demands Legacy integration approaches failed • Costs -- license and services -- exceeded plans • Broker / platform stacks: costly, closed, complex • Infrastructure never scaled to the extent of the enterprise • For example, not one single terminal has opened successfully in over 25 years • The last attempt was Seoul with CORBA – it was a complete failure • Very high development costs, particularly with integration • Time delays where projects are always behind the business needs • Lack of visibility and understanding of systems Typical Result:

13. Service Oriented Architectures lead to: • Reduced costs — Simplifies the integration process by making application interoperability "plug-and-play". By utilising open standards, there is less software infrastructure to purchase and maintain. • Faster time to market — An extended enterprise will be able to respond more quickly to market changes than its competitors, as its business is more agile. • Greater operating efficiencies — Companies will be able to reuse existing application components and utilise new services, rather then manually duplicating them in-house. • Increased customer satisfaction — Through tighter integration of the business value chain and less manual intervention in business processes, suppliers and customers will have greater reuse of data, and more reliable and timely information.

14. BAA’s Enterprise SOA Vision Benefits address long-standing IT dilemmas PORTAL SERVICE USER-DEFINED SERVICE BATCH SYSTEM BROAD-SCALE INTEROPERABILITY APPLICATION SERVER MODULARITY / REUSE RELATIONAL DATABASE INCREMENTAL DEPLOYMENT PROCESS SERVER FLEXIBILITY LEGACY APPLICATION What most people concentrate on are the endpointsBut the problem area is the “white-space” of SOA

15. Architectural Challenges of SOA platform • Dependability – they land planes! • Reliable, high-performance communications between services • High Availability = Business continuity • Security • Flexibility – this is a changing environment • Mechanism to orchestrate process through the network • Ability to dynamically re-configure services to new uses • Ability to normalise in-flight documents between services • Bridge multiple low-level middleware technologies • Reach and scale • Connect any resources regardless of where they are deployed • Scale from initial phases to arbitrarily large deployment • Retain visibility and control of distributed infrastructure

16. Enterprise SOA BAA chose an ESB to address this SOA “white-space” BAA picked an ESB as their SOA framework PORTAL SERVICE USER-DEFINED SERVICE BATCH SYSTEM BROAD-SCALE INTEROPERABILITY APPLICATION SERVER MODULARITY / REUSE RELATIONAL DATABASE INCREMENTAL DEPLOYMENT PROCESS SERVER FLEXIBILITY HIGH AVAILABILITY LEGACY APPLICATION

17. The Purpose of an ESB Connect, Mediate and Control WITHOUT AN ESB WITH AN ESB Connect Control Mediate

18. Connect applications and services Multiple on-ramps, dependable communications • Connect old and new • Legacy applications, RDBMS • J2EE, .Net • Web services • B2B protocols • Link services and processes across the extended enterprise • Establish robust, scalable and secure communications Connect • Examples: • Securely link internal processes with those of business partners. • Portal integration All connected resources are first-class citizens

19. Mediate services Bridge and extend incompatible technologies • Reconcile system incompatibilities • Communication Protocol • Interaction model • Transform and enrich data • Map between data formats • Split, aggregate and enrich data • Provide flexible routing and process flow • Decoupled, event-driven services • Intelligent routing • Support stateful process management • Examples: • Aggregate data from multiple SAP systems • Regulation compliance logging Mediate Eliminate service interdependencies

20. Control service interaction Deploy, configure, manage • Dynamically configure, deploy and upgrade hosted services • Establish and alter process flows, routing, Quality of Service • Gain control and visibility over services and their interaction Control • Examples: • Deploy and upgrade 1000s of end-points from a single location. • Detect faults and diagnose problems in complex deployment. Configured, not coded

21. Global reach, global scalability • ESB spans clusters and security infrastructure to form federated environment • Bus topology obviates hub-and-spoke bottlenecks • Deploy what you need, where and when you need it End-to-end SOA

22. What are the technical problems? The Devil is in the detail…

23. How do you manage a project this big? Business Process Definitions • How do they leverage their existing IT portfolio? • What will this cost? • What would be the impact of • Changes? • Expansion? • New security threats? • Regulation changes? • How will they accommodate future requirements?

24. Imagine Project Managing the Internet How… • …would you scope the project? • …would you consider all future needs • …would you handle training? • …would you manage change? • …would ensure interoperability? • …would you manage the technical differences? • …would you manage the scale? • …would you manage the risk?

25. So how was Internet Successful? What made it work? • Built around a few sacred principles • Evolved from selected technology standards • Deployment abstracted from design • Incremental deployment • Tactical execution

26. Strategy Versus Tactics What makes some ideas work where others fail? • This is how BAA are making T5 a success • Everything is broken down into manageable tasks • Matrix management • Evolutionary project management • NOT Waterfall project management • An IT back-bone and architecture fromthe very start • Industry patterns are being exploited

27. Design patterns • I encourage you to work with patterns – check out: • http://www.enterpriseintegrationpatterns.com/ • http://www.oreilly.com/catalog/esb/index.html • Working with patterns… • Some examples…

28. Scenario from a European Airline Printing

29. Solution Scenarios - CITP

30. CITP in Pattern language

31. Micro Patterns as Services P2P P2P P2P Print Req. JCA JMS MDB Servlet MQ SSB Portlet CITP EJB Web PDS ESB Infrastructure 3 4 2 5 5 5 1 • Print Request arrives at CITP • Request crosses the MQ Series Bridge • Print Token is resolved in PDS • Request is routed via CBR • Request is consumed in Terminal

32. Using a re-factoring pattern XML Integration Broker MQ Series ePoS DB Client Adapter Audit Mainframe SOAP SOAP Browser Adapter Registration App Siebel Agent Reservation Applications Central CRM Finance Applications

33. Incremental Staged Deployment HTTP SOAP WS COM Integration Broker MQ Series Client Adapter ePoS DB Audit Browser Mainframe Enterprise SOA – one step at a time ESB SOAP Adapter Registration App Siebel Agent Reservation Applications Central CRM Finance Applications Service Containers

34. Widely Distributed Enterprises HTTP SOAP WS Integration Broker MQ Series Client Adapter DB Audit ePoS Browser Mainframe ESB ESB SOAP Adapter Registration App Siebel Agent Reservation Applications Central CRM Finance Applications

35. Leave and Layer HTTP SOAP WS Integration Broker MQ Series Client Adapter DB Audit ePoS Browser Mainframe ESB ESB ESB Adapter Registration App Siebel Agent Reservation Applications Central CRM Finance Applications

36. Partner Integration HTTP SOAP WS Integration Broker MQ Series Adapter DB Audit Client Browser ePoS Mainframe ESB ESB ESB SOAP Adapter Registration App Siebel Agent Reservation Applications Central CRM Finance Applications

37. Incremental Adoption Integration Broker Adapter MQ Series DB Audit Client Browser ePoS Mainframe I like to say: “Think Strategically, Act Tactically” Phase n Phase 2 Phase 1 ESB ESB ESB Partner ESB Adapter Registration App Siebel Agent Reservation Applications Central CRM Finance Applications

38. Some architecture tricks to be aware of… Dealing with: Trapped messages Out of Order messages

39. Machine failure causes trapped messages Machine fails! Client 1 2 3 4 Server Two machines, a broker each and clustered

40. Clustering means system carries on straight away Machine fails! Client 1 2 3 4 Server Two machines, a broker each and clustered

41. Machine recovers and messages delivered Machine fails! • Recovery of messages takes: • Machine reboot • OS restart • Software reload/restart • Database recovery • Re-send of messages • = several minutes… Client 1 2 3 4 Server Two machines, a broker each and clustered

42. Can you wait several minutes? • Trader trying to execute a buy • Retailer trying to process your credit card at a till • Bank trying to process mortgages before the end of the day • Airport trying to route baggage • Telco network usage just before billing run at end of month • Many other situations… It gets worse… Now add a requirement for guaranteed message ordering

43. Typical scenario – an airport display board Machine fails! Messages: 1 = BMI256 last call gate 3 2 = BA35 to gate 19 3 = BA35 to gate 23 (correction) 4 = BA765 last call gate 15 1 2 3 4 Flight Information (four messages)

44. Message ordering causes “pile-up” in other broker Machine fails! Due to guaranteed message ordering, can’t deliver other messages! And remember, recovery takes several minutes… (I can see potential for missed flights!) Think of complexity if you add in XA transactions 1 2 Where are my messages! Messages: 1 = BMI256 last call gate 3 2 = BA35 to gate 19 3 = BA35 to gate 23 (correction) 4 = BA765 last call gate 15 3 4 Flight Information (four messages)

45. Turn to Continuous Availability Architecture CAA HOT-HOT deployment Increases overall throughput • Each broker has a hot backup broker on another machine • Transparent to client • Transactional integrity during failover • Once and only once delivery ensured • Fully scalable as we can also add clustering! CAA • Conventional hardware • Seconds to recover • Simple and flexible • Maintains transactional integrity • Easy to use with message ordering

46. Remember our airport? Machine fails! 1 1 2 2 P 3 3 4 4 Secondary becomes primary and sends messages Flight Information (four messages)

47. Overview architecture for BAA T5 Sonic Cluster Sonic Cluster Broker B Broker B Broker B’ Broker B’ Broker A’ Broker A’ Broker A Broker A Sonic ESB custom Javaservices Sonic ESB custom Javaservices Subscribe Subscribe AODB AODB Adapter Adapter ClientApp ClientApp Geographical Site 2 Geographical Site 1 Sonic ESB Sonic ESB

48. Summary – moving to “One Architecture” • I suggest you look into the following: • Evolutionary project management • Things like extreme programming techniques • Design and architecture patterns • www.enterpriseintegrationpatterns.com • Service Oriented Architecture • This is one of the most sought after roles in IT right now • (There were 280 architect roles on monster yesterday) • Enterprise Service Bus concepts – read the ESB book! • SOA has gone past the critical inflection point • Learn SOA, understand the nuances (the detail) • Serious money is being spent on this new technology • Web services is important, but is only one part of the hybrid architecture that companies are moving toward