Fermi National Accelerator Laboratory Don Petravick (PI), Phil Demar, Matt Crawford,

1. Lambda Station: On-demand flow based routing for data intensive GRID applications over multitopology networks Fermi National Accelerator Laboratory Don Petravick (PI), Phil Demar, Matt Crawford, Andrey Bobyshev, Maxim Grigoriev California Institute of Technology. Harvey Newman (co-PI) ESCC/Internet2 Joint Techs Workshop Fermilab, July 15-19, 2007

2. Outline of the talk • goals and major directions of the project • software architecture, API, middleware • results of using Lambda Station service • problems and challenges, plans

3. Basic terms • Lambda Station – a host with special software to control traffic path across LAN and WAN on-demand of applications • PBR – policy based routing • PBR Client – a system or cluster and applications running on it sourcing traffic flows that can be subject for policy based routing • Flow - a stream of packets with some attributes in common such as endpoint IP addresses (or range of addresses), protocols, protocol's ports if applicable and differentiated services code point (DSCP).

4. The goal of the project... deals with the last-mile problem in local area networks The main goal of Lambda Station project is to design, develop and deploy a network path selection services to interface production storage and computing facilities with advanced research networks. • selective forwarding on a per flow basis • alternate network paths for high impact data movement • access control in site edge routers for those selected flows • on-demand from applications (authentication & authorization) • current implementation based on policy-based routing & including the support of DSCP marking

5. Flows and DSCP tagging Any combination of flow's attributes can be used by Lambda Station software to identify flows on per ticket's basis. Typical steps of alternative path reservation: • generate request for service (application, application's proxy, admin) • LS negotiates service and parameters with remote site • configure local and wide area network (not yet available) • marking traffic (if specified). Current LS software is capable to complete all these steps within 3 – 5 mins. That is why it is desirable to know flow selection parameters before transferring is started: • endpoint IP addresses • DSCP

6. LS-Management & Reporting Interface local definitions LS-to-LS service LS-2-LS service online updates Lambda Station Building Blocks Storage & application space Management RemoteLambdaStation SOAP/JClarens SOAP/JClarens LSInterface LS request interface SOAP over HTTPS mySQL:requests, Authorization Service-based Architecture: • LS Controller – manages LS persistence, controls other services • LS persistence is stores current state of the system • LS request Iinterface - webservice for placing all kinds of “ticket” related requests • LS-to-LS – webservice for LS definitions propagation and LS discovery • NETWORK CONFIGURATOR – dynamic reconfiguring of LAN and WAN LS persistence LS Controller NETWORK CONFIGURATOR Vendor specific modules Force10 WAN CISCO Data Exchange Control & Management

7. LS JAVA API Components

8. Java implementation of LS Software • Service Oriented Architecture • utilized JClarens and Axis framework as a web-services toolkit • messages are defined and strongly validated by XML schema • LS service is multi-threaded, one thread for LSController, one thread for LS2LS service and threads pool for openSvcTicket requests • LS2LS and client-2-LS authentication is based on gLite library and supports standard Grid proxies and KCA-issued certificates • Authorization is based on rules set • General framework persistence is accomplished by MySQL DB backend • secure document/literal wrapped SOAP messages, Web Services Interoperability Profile (WS-I Basic Profile Version 1.1)

9. Java implementation of LS Software(continued) • Automated LS and PBR client configuration management • Automated deployment ( one can install on any Linux box) • LS Controller, LS2LS , LS AAA, LS client interface are ready for deployment. Supported java and perl clients. • Some interest from ANL to support C client for Globus toolkit • Network Config calls implemented in interface and may relay requests to perl service ( SOA at work ) • Currently deployed and work ( exchanging PBR and LS configurations) at FNAL ( 2 stations) and CalTech ( 1 station)

10. DSCP Tagging Complexity of using DSCP tagging: • preservation of DSCP is not guaranteed in WAN • DSCP tagging needs to be synchronize between sites for dynamically configurable networks ( asymmetry is bad for high-performance transferring ) LS software does support two different modes of DSCP tagging : • fixed DSCP values to identify site's traffic. • DSCP value is assigned dynamically on per ticket base.

11. Netconfig Module • dynamically modifies the configurations of local network devices. • a vendor dependent components. • Cisco routers with IOS version supporting sequencing type of named ACLs. Tasks to configure PBR in Cisco devices: • interface on which PBR is applied needs to be configured with “ip policy route-map” statement • route map needs to be configured as ordered list of match/action statements • match criteria need to be associated with ACLs

12. openSvcTicket request Accepts many input parameters, most of them could also be determined automatically: • Remote LambdaStation site's identifiers • Source/Destination PBRClient identifier or/and list of IP netblocks • IP protocols, protocol's ports or port range • localPath, remotePath identifiers ( subject to available resources) • Bandwidth out, Bandwidth in • boardTime, startTime, endTime, travelTime Returns ID of locally assigned ticket Operational modes (subject to authentication, authorization): • new ticket - creating a new ticket • join - provides ID and required parameters(DSCP) of already opened ticket • extend ticket – allows to extend already created ticket.

13. 4 3 2 1 LSiperf End-to-End Test • Data transfer started: • 10GE host; 5 tcp streams • Network path is via ESnet • OC12 bottleneck… • Path MTU is 1500B • LambdaStation service ticket is opened • LambdaStation changes network path to USN • Host path MTUD check detects a larger path MTU • LambdaStation service ticket expires: • Network path changed back to ESnet

14. Lambda Station TestBed

15. normal traffic flow High Impact traffic SRM/DCache 1.7 LS-awareness CalTech Advanced Networks Wide Area Network SRM CalTech LambdaStation StarLight FNAL LambdaStation CMS core router Site Network Production USCMS SRM server sends requests to Lambda Station to stir a high-impact traffic into Advanced Network infrastructure CMS SRM How to demonstrate increasing transfer rate ?? In tests we could generate a data stream and monitor its rates when switching traffic between alternative paths. Production traffic is not deterministic. USCMS Tier1

16. Project accomplishments • Software version 1.0 (a fully functional prototype supporting whole cycle of LS functionality) • positive results of testing between Fermilab and Caltech • lsiperf, lsTraceroute – wrappers around well known applications to add Lambda Station awareness ( based on prototype version 1.0) • SRM/dCache integration – testing, added in production 1.7.0 release • run LS-aware SRM/dCache on production cluster at Fermilab • Interoperable Java implementation of the major components of LambdaStation ( perl, java clients available)

17. Problems and challenges • Traffic Asymmetry is bad for high performance applications • Network (Lambda Station) awareness is too complex • Definition of PBR Clients is a complex issue, auto definition is not yet available Plans • release fully functional Java LS • SRM/dCache with production quality LS support • add real-time monitoring of resources ( perfSONAR ) • add WAN control plane module • integration with OSCARS ( unified Network Path Reservation Model ?) References: http://www.lambdastation.org/

18. END

19. Miscellaneous Slides • Simulation of multiple Lss • screen shoots of ticket's queue • SC05 Demo

21. Netconfig Module • dynamically modifies the configurations of local network devices. • a vendor dependent components. • Cisco routers with IOS version supporting sequencing type of named ACLs. Tasks to configure PBR in Cisco devices: • interface on which PBR is applied needs to be configured with “ip policy route-map” statement • route map needs to be configured as ordered list of match/action statements • match criteria need to be associated with ACLs

22. openSvcTicket request Accepts many input parameters, most of them could also be determined automatically: • Dst site's identifiers • Src/Dst PBRClient identifier or/and list of Src/Dst IP in (CIDR) • IP protocols, protocol's ports • localPath, remotePath identifiers • BWout,BWin • boardTime, startTime, endTime Returns ID of locally assigned ticket Operational modes (subject authentication, authorization and quoting): • new ticket - creating a new ticket • join - provides ID and required parameters(DSCP) of already opened ticket • extend ticket – allows to extend already created ticket.

23. Directions of the project. • building a wide-are testbed infrastructure • designing, developing Lambda Station software, Lambda Station (network) aware applications • researching effects of flow based switching on applications

24. Application's behavior in flow based switching environment Tuning of end systems for maximum rates is not a subject of the Lambda Station project, however, we need to see an increase of data movement performance when selectively switch flows • DSCP tagging with IPtables • switching between two paths with different MTUs

25. Effect of DSCP tagging with IPTables

26. Motivation of the project • unprecedented demands for data movement in physics experiments such as CDF, D0, BarBar and coming LHC experiments • massive, globally distributed datasets growing to the 100 petabytes by 2010 • collaborative data analysis by global communities of thousands of scientists • available data communication technology will not be able to satisfy these demands simply by plain increasing bandwidth in LANs and WANs due to technology limitations and high deployment and operation costs. • Advance Research optical networks – greater capacity, no production quality of service, are not universally available for all pairs of communicating endpoints

27. NG-ADM Multiple Network Toplogies Admission Group of network devices Blue NG-B Green RT1 RT3 RT2 Red PBR-client RED-B-IN PBR-clients or regular clients at the remote sites RED-ClientA-IN GREEN-ClientB-IN RED-ClientB-IN GREEN-ClientB-IN NG-C Client A, RED & GREEN rules for NGC RT1 RT3 RT2 BLUE is Production Path ClientA rules for Red& GREEN topologies: RED-OUT GREEN-OUT NG-A RT1 RT3 RT2 H3 H1 H1 H2 H3 H2 PBR-client B PBR-client A LS multitopology network model Cisco IOS, dynamic configuring of PBR, extended sequencing ACLs + access policy ACLs

28. LambdaStation SC05 Demo Fermilab SC05/Seattle Commodity Internet/SCinet lambdastation@FNAL lambdastation@SC05 SC05/HighSpeed Links nws-lab.fnal.gov A122.302.sc05.org lsiperf A126.302.sc05.org srmcp charley.fnal.gov

30. PMTUD Note A Note A: We believe it is a HW/ASIC problem with SNMP monitoring, a time to time SNMP -get returns the same counters as in previous cycle.