1 / 30

US Grid Efforts

US Grid Efforts. Lee Lueking D0 Remote Analysis Workshop February 12, 2002.

lynton
Download Presentation

US Grid Efforts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. US Grid Efforts Lee Lueking D0 Remote Analysis Workshop February 12, 2002

  2. All of these projects are working towards the common goal of providing transparent access to the massively distributed computing infrastructure that is needed to meet the challenges of modern experiments … (From the EU DataTAG proposal) Lee Lueking - D0 RACE

  3. Grid Projects Timeline Q3 00 GriPhyN: $11.9M+$1.6M Q4 00 EU DataGrid: $9.3M Q1 01 Q2 01 PPDG:$9.5M Q3 01 EU DataTAG:4M Euros iVDGL:$13.65M Q4 01 GridPP: Q1 02 Lee Lueking - D0 RACE

  4. PPDG • Develop, acquire and deliver vitally needed Grid-enabled tools for data-intensive requirements of particle and nuclear physics. • Collaboration of computer scientists with a strong record in distributed computing and Grid technology, and physicists with leading roles in the software and network infrastructures for major high-energy and nuclear experiments. • Goals and plans are ultimately guided by the immediate, medium-term and longer-term needs and perspectives of the physics experiments. Lee Lueking - D0 RACE

  5. GriPhyN: Grid Physics Network • Virtual data technologies. Advances are required in information models and in new methods of cataloging, characterizing, validating, and archiving software components to implement virtual data manipulations • Policy-driven request planning and scheduling of networked data and computational resources. We require mechanisms for representing and enforcing both local and global policy constraints and new policy-aware resource discovery techniques. • Management of transactions and task-execution across national-scale and worldwide virtual organizations. New mechanisms are needed to meet user requirements for performance, reliability, and cost. Lee Lueking - D0 RACE

  6. iVDGL:International Virtual Data Grid Laboratory • The iVDGL will provide a global computing resource for several leading international experiments in physics and astronomy, • Global services and centralized monitoring, management, and support functions functions will be coordinated by the Grid Operations Center (GOC) located at Indiana University, with technical effort provided by GOC staff, iVDGL site staff, and the CS support teams. • GriPhyN and Particle Physics Data Grid will provide the basic R&D and software toolkits needed for the laboratory. • The European Union DataGrid is also a major participant and will contribute basic technologies and tools. • The iVDGL will be based on the open Grid infrastructure provided by the Globus Toolkit and will also build on other technologies such as Condor resource management tools. Lee Lueking - D0 RACE

  7. Comparison of PPDG and iVDGL Lee Lueking - D0 RACE

  8. PPDG Collaborators Lee Lueking - D0 RACE

  9. PPDG Computer Science Groups Condor – develop, implement, deploy, and evaluate mechanisms and policies that support High Throughput Computing on large collections of computing resources with distributed ownership. http://www.cs.wisc.edu/condor/ Globus - developing fundamental technologies needed to build persistent environments that enable software applications to integrate instruments, displays, computational and information resources that are managed by diverse organizations in widespread locations http://www.globus.org/ SDM - Scientific Data Management Research Group – optimized and standardized access to storage systems http://gizmo.lbl.gov/DM.html Storage Resource Broker - client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and cataloging/accessing replicated data sets. http://www.npaci.edu/DICE/SRB/index.html Lee Lueking - D0 RACE

  10. Delivery of End-to-End Applications& Integrated Production Systems to allow thousands of physicists to share data & computing resources for scientific processing and analyses • PPDG Focus: • Robust Data Replication • - Intelligent Job Placement • and Scheduling • - Management of Storage • Resources • - Monitoring and Information • of Global Services • Relies on Grid infrastructure: • - Security & Policy • High Speed Data Transfer • - Network management Operators & Users Resources: Computers, Storage, Networks Lee Lueking - D0 RACE

  11. Common Services • Job Description Language • Scheduling and Management of Processing and Data Placement Activities • Monitoring and Status Reporting • Storage Resource Management • Reliable Replica Management Services • File Transfer Services • Collect and Document Current Experimental Practices • R & D, Evaluation • Authentication, Authorization, and Security • End-to-End Applications and Testbeds Lee Lueking - D0 RACE

  12. Project Activities,End-to-End Applicationsand Cross-Cut Pilots • Project Activities are focused Experiment – Computer Science Collaborative developments. • Replicated data sets for science analysis – BaBar, CMS, STAR • Distributed Monte Carlo production services – ATLAS, D0, CMS • Common storage management and interfaces – STAR, JLAB • End-to-End Applications used in Experiment data handling systems to give real-world requirements, testing and feedback. • Error reporting and response • Fault tolerant integration of complex components • Cross-Cut Pilots for common services and policies • Certificate Authority policy and authentication • File transfer standards and protocols • Resource Monitoring – networks, computers, storage. Lee Lueking - D0 RACE

  13. Security, Privacy, Legal Super Computing 2001 in Denver Lee Lueking - D0 RACE

  14. Lee Lueking - D0 RACE

  15. PPDG activities as part of the Global Grid Community • Coordination with other Grid Projects in our field: • GriPhyN – Grid for Physics Network • European DataGrid • Storage Resource Management collaboratory • HENP Data Grid Coordination Committee • Participation in Experiment and Grid deployments in our field: • ATLAS, BaBar, CMS, D0, Star, JLAB experiment data handling systems • iVDGL/DataTAG – International Virtual Data Grid Laboratory • Use DTF computational facilities? • Active in Standards Committees: • Internet2 HENP Working Group • Global Grid Forum Lee Lueking - D0 RACE

  16. PPDG and GridPP Projects • Use of Standard Middleware to Promote Interoperability • Move to Globus infrastructure: GSI, GridFTP • Use of Condor as a supported system for job submission • Publish availability of resources and file catalog • Additional Grid Functionality for Job Specification, Submission, and Tracking • Use Condor for migration and check pointing • Enhanced job specification language and services • Enhanced Monitoring and Diagnostic Capabilities • Fabric Management Lee Lueking - D0 RACE

  17. PPDG Management and Coordination PIsLivny, Newman, Mount Steering CommitteeRuth Pordes, ChairDoug Olson, Physics Deputy ChairMiron Livny, Computer Science Deputy ChairComputer Science Group RepresentativesPhysics Experiment RepresentativesPIs (ex officio) Executive Team (>1.0 FTE on PPDG)Steering Committee ChairSteering Committee Physics and CS Deputy Chairs STAR SDM BaBar SRB JLAB ATLAS Globus CMS Condor DZero Lee Lueking - D0 RACE

  18. iVDGL “We propose to create, operate and evaluate, over asustained period of time, an international researchlaboratory for data-intensive science.” From NSF proposal, 2001 • International Virtual-Data Grid Laboratory • A global Grid laboratory with participation from US, EU, Asia, etc. • A place to conduct Data Grid tests “at scale” • A mechanism to create common Grid infrastructure • A facility to perform production exercises for LHC experiments • A laboratory for other disciplines to perform Data Grid tests Lee Lueking - D0 RACE

  19. iVDGL Summary Information • Principal components (as seen by USA) • Tier1 sites (laboratories) • Tier2 sites (universities and other institutes) • Selected Tier3 sites (universities) • Fast networks: US, Europe, transatlantic • International Grid Operations Center (iGOC) • Computer Science support teams • Coordination, management • Proposed international partners • Initially US, EU, Japan, Australia • Other world regions later • Discussions w/ Russia, China, Pakistan, India, South America • Complementary EU project: DataTAG • Transatlantic network from CERN to STAR-TAP (+ people) • Initially 2.5 Gb/s Lee Lueking - D0 RACE

  20. US Proposal to NSF • US proposal approved by NSF Sept. 25, 2001 • “Part 2” of GriPhyN project • Much more application oriented than first GriPhyN proposal • $15M, 5 years @ $3M per year (huge constraint) • CMS + ATLAS + LIGO + SDSS/NVO + Computer Science • Scope of US proposal • Deploy Grid laboratory with international partners • Acquire Tier2 hardware, Tier2 support personnel • Integrate of Grid software into applications • CS support teams (+ 6 UK Fellows) to harden tools • Establish International Grid Operations Center (iGOC) • Deploy hardware at 3 minority institutions (Tier3) Lee Lueking - D0 RACE

  21. US iVDGL Proposal Participants • U Florida CMS • Caltech CMS, LIGO • UC San Diego CMS, CS • Indiana U ATLAS, iGOC • Boston U ATLAS • U Wisconsin, Milwaukee LIGO • Penn State LIGO • Johns Hopkins SDSS, NVO • U Chicago CS • U Southern California CS • U Wisconsin, Madison CS • Salish Kootenai Outreach, LIGO • Hampton U Outreach, ATLAS • U Texas, Brownsville Outreach, LIGO • Fermilab CMS, SDSS, NVO • Brookhaven ATLAS • Argonne Lab ATLAS, CS T2/Software CS support T3/Outreach T1/Labs Lee Lueking - D0 RACE

  22. iVDGL Partners • National partners • PPDG (Particle Physics Data Grid ) • DTF: Distributed Terascale Facility • CAL-IT2 (new California Grid initiative) • Current international partners • EU-DataGrid • UK PPARC funding agency • UK Core e-Science Program  6 UK Fellowships • INFN (Italy) • 2 Japanese institutes • 1 Australian institute (APAC) Lee Lueking - D0 RACE

  23. Tier0/1 facility Tier2 facility Tier3 facility 10 Gbps link 2.5 Gbps link 622 Mbps link Other link iVDGL Map Circa 2002-2003 Lee Lueking - D0 RACE

  24. iVDGL Requirements • Realistic scale • In number, diversity, distribution, network connectivity • Delegated management and local autonomy • Management needed to operate as large, single facility • Autonomy needed for sites and experiments • Support large-scale experimentation • To provide useful information for building real Data Grids • Robust operation • For long running applications in complex environment • Instrumentation and monitoring • Required for an experimental facility • Integration with international “cyberinfrastructure” • Extensibility Lee Lueking - D0 RACE

  25. Approach • Define a laboratory architecture • Define expected laboratory functions • Build in scalability, extensibility, reproducibility • Define instrumentation, monitoring • Establish CS support teams (develop/harden tools, support users) • Define working relationship, coordination with partners • Create and operate global-scale laboratory • Deploy hardware, software, personnel at Tier2, Tier3 sites • Establish iGOC, single point of contact for monitoring, support, … • Help international partners establish sites • Evaluate and improve iVDGL through experimentation • CS support teams will work with experiments • Extend results to partners • Engage underrepresented groups • Integrate minority institutions as Tier3 sites Lee Lueking - D0 RACE

  26. iVDGL as a Laboratory • Grid Exercises • “Easy”, intra-experiment tests first (10-30%, national, transatlantic) • “Harder” wide-scale tests later (30-100% of all resources) • CMS is already conducting transcontinental simulation productions • Operation as a facility • Common software, central installation to ensure compatibility • CS teams to “harden” tools, support applications • iGOC to monitor performance, handle problems Lee Lueking - D0 RACE

  27. Emphasize Simple Operation • “Local” control of resources vitally important • (Site level or national level) • Experiments, politics demand it • Operate mostly as a “partitioned” testbed • (Experiment, nation, etc.) • Avoids excessive coordination • Allows software tests in different partitions • Hierarchy of operation must be defined • E.g., (1) National + experiment, (2) inter-expt., (3) global tests Lee Lueking - D0 RACE

  28. Other Disciplines • Use by other disciplines • Expected to be at the 10% level • Other HENP experiments • Virtual Observatory (VO) community in Europe/US • Gravity wave community in Europe/US/Australia/Japan • Earthquake engineering • Bioinformatics • Our CS colleagues (wide scale tests) Lee Lueking - D0 RACE

  29. US iVDGL Management and Coordination Collaboration Board (Advisory) External Advisory Board Project DirectorsAvery, Foster Project Coordination GroupProject CoordinatorProject DirectorsCoordinators of Systems Integration, Education/OutreachPhysics Experiment RepresentativesUniversity Research Center or Group RepresentativesPACI Representatives iVDGL Design and Deployment Integration with Applications University Research Centers / Groups International Grid Operations Center Lee Lueking - D0 RACE

  30. Conclusion • PPDG, and iVDGL are complementary in their approach and deliverables. • These efforts, along with our European partners will provide exciting new ways to share data and computing resources. • Dzero Grid involvement offers many challenges, but even more opportunities. • Acknowledgements: Richard Mount (SLAC), Paul Avery (University of Florida), Ruth Pordes (FNAL). Lee Lueking - D0 RACE

More Related