The Project : Enabling Data Intensive Science at Universities

1. The Project : Enabling Data Intensive Science at Universities Collaboration Caltech (lead inst.) BNL Michigan MIT Florida Florida International FNAL San Diego SLAC Vanderbilt Shawn McKee for R. Cavanaugh University of Florida T3 Meeting Univ. of Michigan

2. UltraLight: A New Class of Integrated Information Systems • Delivering the next generation of network-aware real-time Grids • The network as an integrated, managed resource • Hybrid packet-switched + dynamic optical paths • Leveraging Trans-US, Transatlantic & Transpacific network partnerships; • With ESnet, USNet, KEK, Kreonet, GLORIAD, CHEPREO, WHREN/LILA, Awave, FLR, Pacific Wave, Translight, Netherlight • Extensions to Korea, Brazil, Japan and Taiwan • MONALISA/VINCI: End-to-end monitoring, tracking, dynamic BW provisioning and workflow optimization

3. UltraLight Network Laboratory Testbed & Facility

4. What is Achievable in the Lab? Rate [Gbs] Example: • Supercomputing 2005 • 151 Gbps peak rate • 100+ Gbps sustained throughput for hours • 475 Terabytes of physics data transported in less than 24 hours • Sustained rate of 100+ Gb/s translates to > 1 Petabyte per day 0 15 30 45 60 t [min] Cumulative [TB] 0 6 12 18 24 t [hours]

5. FDT – Fast Data Transport A New Application for Efficient Data Transfers • Capable of reading and writing at disk speed over wide area networks (with standard TCP) for the first time • Highly portable and easy to use: runs on all major platforms. • Based on an asynchronous, flexible multithreaded system, using the Java NIO libraries, that: • Streams a dataset (list of files) continuously, from a managed poolof buffers in kernel space, through an open TCP socket • Smooth flow of data from each disk • No protocol start phase between files • Uses independent threads to read and write on each physical device • Transfers data in parallel on multiple TCP streams, when necessary • Uses appropriate-sized buffers for disk I/O and for the network • Restores the files from buffers asynchronously • Resumes a file transfer session without loss, when needed • GSI Authentication built in

6. Major Communities OSG CMS ALICE D0 STAR VRVS LGC RUSSIA SE Europe GRID APAC Grid UNAM Grid (Mx) ITU ABILENE ULTRALIGHT GLORIAD LHC Net RoEduNET Enlightened VRVS ALICE MonALISA Monitoring System • MonALISA Today • Running 24 X 7 at 280 Sites • Collecting > 750,000 parameters in near real-time • Update rate of 25,000 parameter-updates per sec • Monitoring • 40,000 computers • > 100 WAN Links • Thousands of Grid jobs running concurrently • Demonstrated at: • Telecom World • WSIS 2003 • SC 2004 • Internet2 2005 • TERENA 2005 • IGrid 2005 • SC 2005 • CHEP 2006 • CENIC 2006 Innovation Award for High-Performance Applications ABILENE - - OSG VRVS

7. L-Store: File System Interface to Global Storage • Provides a file system interface to (globally) distributed storage devices (“depots”) • Parallelism for high performance and reliability • Uses IBP (from UTenn) for data transfer & storage service • Generic, high performance, wide-area-capable storage virtualization service; transport plug-in support • Write: break file into blocks, upload blocks simultaneously to multiple depots (reverse for reads) • Multiple metadata servers increase performance & fault tolerance • L-Store supports beyond-RAID6-equivalent encoding of stored files for reliability and fault tolerance • Supports scalable streaming of Physics Analysis Data in/out of L-Store over the WAN

8. and VINCI Services (Virtual Intelligent Network Computing Infrastructure) Application Application End User Agent End User Agent System Evaluation & Optimization Authentication, Authorization, Accounting Scheduling; Dynamic Path Allocation Prediction Failure Detection Topology Discovery Control Path Provisioning Learning SNMP GMPLS MPLS OS MonALISA Monitoring Substrate

9. VINCI: Dynamic Path Allocation for Automated Dataset Transfer Detects errors & automatically recreates the path in less than the TCP session timeout (<1second) >MLcopy A/fileX B/path/ OS path available Configuring interfaces Starting Data Transfer Normal Routed Path Real time monitoring Regular IP path APPLICATION DATA MonALISA Services A Monitor Control B TL1 LISA AGENT Sets up • Network Interfaces • TCP stack • Kernel parameters • Routes LISA Agent Optical Switch Active light paths

10. Physicist at Tier3 using Root on GBytes of ntuples Loads Clarens Root plugin. Connects to Clarens. Sends analysis code (.C/.h files). Clarens creates Rootlet, passes it .C/.h files Rootlet runs analysis code on TBytes of ntuples, creating high statistics output data. Root at Tier3 receives and plots data “Rootlets” Root embedded in a Clarens server Tier3 Tier2 Analysis.C, Analysis.h    Clarens Plugin XML/RPC Root Tuples  Root Tuples GBytes  ~10 TBytes

11. “Higgs” Data Analysis using Rootletsat Supercomputing ’06 Bandwidth Challenge

12. Summary: UltraLight Network R&DGlobal Planning Services • UltraLight is a global Laboratory, uniquely positioned • Spans Tier-0, some Tier-1s, several Tier-2s, and Tier-3s • UltraLight (optical networks in general) moving towards a managed “control plane” • Light-paths will be allocated/scheduled to data-flow requests via policy based priorities, queues, and advanced reservations • Clear need to match “Network Resource Management” with “Storage Resource Management” • Available UltraLight Infrastructure / Services / Applications for Tier3s • High performance network infrastructure • Fast Data Transport Tool – FDT • MonALISA + VINCI • LStore • Rootlets • UltraLight working to develop (generic) systems solutions by • Researching and developing Global Planning Services • Using an end-to-end approach • devices, parameters, end-host services, WAN, high-level services, applications Services fabric level application level