1 / 26

HEP: Networks, Grids and Collaborative Systems for Global VOs

HEP: Networks, Grids and Collaborative Systems for Global VOs. Harvey B. Newman for the Caltech SLAC LANL Team SC2003 Bandwidth Challenge, Phoenix November 19, 2003. Next Generation Networks and Grids for HEP Experiments.

tallis
Download Presentation

HEP: Networks, Grids and Collaborative Systems for Global VOs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HEP: Networks, Grids and Collaborative Systems for Global VOs Harvey B. Newman for the Caltech SLAC LANL Team SC2003 Bandwidth Challenge, PhoenixNovember 19, 2003

  2. Next Generation Networks and Grids for HEP Experiments • Providing rapid access to event samples and analyzed physics results drawn from massive data stores • From Petabytes in 2003, ~100 Petabytes by 2007-8, to ~1 Exabyte by ~2013-5. • Providing analyzed results with rapid turnaround, bycoordinating and managing large but LIMITED computing, data handling and NETWORKresources effectively • Enabling rapid access to the Data and the Collaboration • Across an ensemble of networks of varying capability • Advanced integrated applications, such as Data Grids, rely on seamless operation of our LANs and WANs • With reliable, monitored, quantifiable high performance Worldwide Analysis: Data explored and analyzed by thousands of globally dispersed scientists, in hundreds of teams

  3. Large Hadron Collider (LHC) CERN, Geneva: 2007 Start • pp s =14 TeV L=1034 cm-2 s-1 • 27 km Tunnel in Switzerland & France CMS TOTEM pp, general purpose; HI First Beams: April 2007 Physics Runs: from Summer 2007 ALICE : HI LHCb: B-physics Atlas ATLAS Data Challenges; Computing and Physics TDRs 2004-5

  4. Four LHC Experiments: The Petabyte to Exabyte Challenge • ATLAS, CMS, ALICE, LHCBHiggs + New particles; Quark-Gluon Plasma; CP Violation Data stored Tens of PB 2008; To 1 EB by ~2015 CPU Hundreds of TFlopsto PetaFlops

  5. LHC: Higgs Decay into 4 muons (Tracker only); 1000X LEP Data Rate 109 events/sec, selectivity: 1 in 1013 (1 person in a thousand world populations)

  6. CERN/Outside Resource Ratio ~1:2Tier0/( Tier1)/( Tier2) ~1:1:1 ~PByte/sec ~100-1500 MBytes/sec Online System Experiment CERN Center PBs of Disk; Tape Robot Tier 0 +1 Tier 1 ~2.5-10 Gbps FNAL Center IN2P3 Center INFN Center RAL Center 2.5-10 Gbps Tier 2 Tier2 Center Tier2 Center Tier2 Center Tier2 Center Tier2 Center ~2.5-10 Gbps Tier 3 Institute Institute Institute Institute Tens of Petabytes by 2007-8.An Exabyte ~5-7 Years later. Physics data cache 0.1 to 10 Gbps Tier 4 Workstations LHC Data Grid Hierarchy:Developed at Caltech Emerging Vision: A Richly Structured, Global Dynamic System

  7. Production BW Growth of Int’l HENP Network Links (US-CERN Example) • Rate of Progress >> Moore’s Law. (US-CERN Example) • 9.6 kbps Analog (1985) • 64-256 kbps Digital (1989 - 1994) [X 7 – 27] • 1.5 Mbps Shared (1990-3; IBM) [X 160] • 2 -4 Mbps (1996-1998) [X 200-400] • 12-20 Mbps (1999-2000) [X 1.2k-2k] • 155-310 Mbps (2001-2) [X 16k – 32k] • 622 Mbps (2002-3) [X 65k] • 2.5 Gbps (2003-4) [X 250k] • 10 Gbps  (2005) [X 1M] • A factor of ~1M over a period of 1985-2005 (a factor of ~5k during 1995-2005) • HENP has become a leading applications driver, and also a co-developer of global networks

  8. HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps Continuing the Trend: ~1000 Times Bandwidth Growth Per Decade;We are Rapidly Learning to Use Multi-Gbps Networks Dynamically

  9. HEP is Learning How to Use Gbps Networks Fully: Factor of ~50 Gain in Max. Sustained TCP Thruput in 2 Years, On Some US+Transoceanic Routes • 9/01 105 Mbps 30 Streams: SLAC-IN2P3; 102 Mbps 1 Stream CIT-CERN • 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams • 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel) • 9/02 850, 1350, 1900 Mbps Chicago-CERN 1,2,3 GbE Streams, 2.5G Link • 11/02 [LSR] 930 Mbps in 1 Stream California-CERN, and California-AMS FAST TCP 9.4 Gbps in 10 Flows California-Chicago • 2/03 [LSR] 2.38 Gbps in 1 Stream California-Geneva (99% Link Utilization) • 5/03 [LSR] 0.94 Gbps IPv6 in 1 Stream Chicago- Geneva • TW & SC2003: 5.65 Gbps (IPv4), 4.0 Gbps (IPv6) in 1 Stream Over 11,000 km *

  10. Fall 2003: Ultraspeed TCP Data Stream Across the Atlantic; 10 GbE over 11,000 km • Terabyte Transfers by the Caltech-CERN Team: • Nov 18: 4.00 Gbps IPv6 Geneva-Phoenix (11.5 kkm) • Oct 15: 5.64 Gbps IPv4 Palexpo-L.A. (10.9 kkm) • Across Abilene (Internet2) Chicago-LA, Sharing with normal network traffic • Peaceful Coexistence with a Joint Internet2- Telecom World VRVS Videoconference European Commission Juniper,Level(3)Telehouse 10GigE NIC

  11. “World Laboratory Experiment” BW Challenge Performance Summary • Utilized all Three Wavelengths, from Three Booths:Caltech, SLAC/FNAL, Los Alamos; + CERN, U. Manchester, U. Amsterdam • Traffic to or from: CENIC, Caltech, SLAC/Palo Alto, TeraGrid, Starlight, Netherlight/UvA, Georgia Tech, CERN/Geneva, and Tokyo • ONLY TCP traffic (FAST (primary), RENO, HSTCP, Scalable) • High speed data transfer application : Clarens, Grid-Enabled Analysis • Disk to Disk transfers: • Two Streams of 200 MB/s each between LA (Cenic PoP) and Phoenix • Memory to memory transfer • Not enough CPU resources to run more disk to disk transfers • Link utilization (Typical Single Streams 4-5 Gbps) • Max. rate on CENIC’s Wave: 10.0 Gbps (output) • Max. rate on Abilene’s Wave: 8.7 Gbps (output) • Max. rate on TeraGrid’s Wave: 9.6 Gbps (input) • Sponsors: DOE, NSF, SCINet, Cisco, Level(3), Nortel, Starlight, CENIC, Internet2, NLR, Intel, HP, ASCI • Servers Used: Dual Itanium2, Opteron, Xeon

  12. Bandwidth Challenge:Network MAP

  13. ns-2 simulation Main data transport protocol used for the bandwidth challenge: FAST TCP • Reno TCP has poor performance w/largewindows • FAST: • Uses end-to-end delay and loss • Very high link utilization (>90% in theory) • Sender side modification only • Fast convergence to equilibrium • Achieves any desired fairness, expressed by a utility function • Pacing: reducing burstiness 95% 1G average utilization 27% 10Gbps 19% txq=100 txq=100 txq=10000 Linux TCP Linux TCP FAST capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 20Gbps; 100 ms round trip latency; 100 flows J. Wang (Caltech, June 02) Capacity = 1Gbps; 180 ms round trip latency;1 flow C. Jin, D. Wei, S. Ravot, etc (Caltech, Nov 02)

  14. Bandwidth Challenge:Data Intensive Distributed Analysis • Analysis of particle collision events recorded by the CMS detector looks for needle in very large haystack • For challenge event, use simulated events produced during a large distributed production run • Multiple archive files with 767,000 events each stored on Clarens servers at CENIC POP in LA, and TeraGrid node at Caltech. • Transferred > 200 files at rates up to 400MB/s to 2 disk servers on the show floor • Decomposed archive files into ROOT data files, published via Clarens on disk servers • Analysis of data performed and results displayed by Clarens ROOT client on kiosk machine

  15. Grid Enabled Analysis: View of a Collaborative Desktop • Building the GAE is the “Acid Test” for Grids; and is crucial for next-generation experiments at the LHC • Large, Diverse, Distributed Community of users • Support hundreds to thousands of analysis tasks, shared among dozens of sites • Widely varying task requirements and priorities • Need Priority Schemes, robust authentication and Security • Relevant to the future needs of research and industry External Services Storage Resource Broker CMS ORCA/COBRA Browser MonaLisa Iguana ROOT Cluster Schedulers PDA ATLAS DIAL Griphyn VDT Clarens VO Management File Access MonaLisa Monitoring Authentication Key Escrow Shell Authorization Logging

  16. “Private” Grids”: Structured P2PSub-Communities in Global HEP

  17. HENP Lambda Grids:Fibers for Physics • Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data Stores • Survivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007)requires that each transaction be completed in a relatively short time. • Example: Take 800 secs to complete the transaction. Then Transaction Size (TB)Net Throughput (Gbps) 1 10 10 100 100 1000 (Capacity of Fiber Today) • Summary: Providing Switching of 10 Gbps wavelengthswithin ~2-4 years; and Terabit Switching within 5-8 years would enable “Petascale Grids with Terabyte transactions”,to fully realize the discovery potential of major HENP programs, as well as other data-intensive fields.

  18. HEP Grid Challenges: Workflow Management and Optimization • Maintaining a Global View of Resources and System State • End-to-end Monitoring • Adaptive Learning: New paradigms for optimization, problem resolution (progressively automated) • Balancing Policy Against Moment-to-moment Capability • Balance High Levels of Usage of Limited Resources Against Better Turnaround Times for Priority Jobs • Realtime Error Detection, Propagation; Recovery • An Integrated User Environment • User-Grid Interactions • Emerging Strategies and Guidelines • Including the Network as a Dynamic, Managed Resource

  19. Lookup Discovery Service Lookup Service Service Listener Lookup Service Remote Notification Registration Station Server Station Server Station Server Proxy Exchange Dynamic Distributed Services Architecture (DDSA) • “Station Server” Services-engines at sites host “Dynamic Services” • Auto-discovering, Collaborative • Scalable to thousands of service-Instances • Servers interconnect dynamically; form a robust fabric • Service Agents: Goal-Oriented, Autonomous, Adaptive • Maintain State: Automatic“Event” notification • Adaptable to Web services, OGSA: many platforms & working environments (also mobile) See http://monalisa.cacr.caltech.edu http://diamonds.cacr.caltech.edu Caltech/UPB (Romania)/NUST (Pakistan) Collaboration

  20. California Institute of Technology

  21. Monitoring CMS farms and WAN traffic

  22. Global Client / Dynamic Discovery Monitoring & Managing VRVS Reflectors

  23. SEA POR SAC NYC CHI OGD DEN SVL CLE PIT WDC FRE KAN RAL NAS National Lambda Rail STR PHO LAX WAL ATL SDG OLG DAL JAC UltraLight Collaboration:http://ultralight.caltech.edu • Caltech, UF, FIU, UMich, SLAC,FNAL,MIT/Haystack,CERN, UERJ(Rio), NLR, CENIC, UCAID,Translight, UKLight, Netherlight, UvA, UCLondon, KEK, Taiwan • Cisco, Level(3) • Integrated packet switched and circuit switched hybrid experimental research network; leveraging transoceanic R&D network partnerships • NLR Waves: 10 GbE (LAN-PHY) wave across the US • Optical paths transatlantic; extensions to Japan, Taiwan, Brazil • End-to-end monitoring; Realtime tracking and optimization; Dynamic bandwidth provisioning, • Agent-based services spanning all layers of the system, from the optical cross-connects to the applications.

  24. UltraLight http://ultralight.caltech.edu • Serving the major LHC experiments; developments broadly applicable to other data-intensive programs • “Hybrid” packet-switched and circuit-switched, dynamically managed optical network • Global services for system management • Trans-US wavelength riding on NLR: LA-SNV-CHI-JAX • Leveraging advanced research & production networks • USLIC/DataTAG, SURFnet/NLlight, UKLight, Abilene, CA*net4 • Dark fiber to CIT, SLAC, FNAL, UMich; Florida Light Rail • Intercont’l extensions: Rio de Janeiro, Tokyo, Taiwan • Flagship Applications with a diverse traffic mix • HENP: TByte to PByte “block” data transfers at 1-10+ Gbps • eVLBI: Real time data streams at 1 to several Gbps

  25. KEK (JP) VRVS (Version 3) Meeting in 8 Time Zones VRVS on Windows Caltech (US) RAL (UK) Brazil CERN (CH) AMPATH (US) Pakistan SLAC (US) Canada 78 Reflectors Deployed Worldwide Users in 96 Countries AMPATH (US)

  26. HENP Networks and Grids; UltraLight • The network backbones and major links used by major HENP projects are advancing rapidly • To the 10 G range in 18 months; much faster than Moore’s Law • Continuing a trend: a factor ~1000 improvement per decade • Transition to a community-owned and operated infrastructure for research and education is beginning with (NLR, USAWaves) • HENP is learning to use 10 Gbps networks effectively over long distances • Fall 2003 Development: 5 to 6 Gbps flows over 11,000 km • A new HENP and DOE Roadmap: Gbps to Tbps links in ~10 Years • UltraLight: A hybrid packet-switched and circuit-switched network: ultra-protocols (FAST), MPLS + dynamic provisioning • To serve the major needs of the LHC & Other major programs • Sharing, augmenting NLR and internat’l optical infrastructures • A cost-effective model for future HENP, DOE networks

More Related