1 / 40

Virtualisation and its Application on the Grid

Virtualisation and its Application on the Grid. Institut für Wissenschaftliches Rechnen Forschungszentrum Karlsruhe Institut für Experimentelle Kernphysik Universität Karlsruhe Volker Büge , Yves Kemp, Marcel Kunze, Günter Quast, Armin Scheurer. Outline.

wallyl
Download Presentation

Virtualisation and its Application on the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virtualisation and its Application on the Grid Institut für Wissenschaftliches Rechnen Forschungszentrum Karlsruhe Institut für Experimentelle Kernphysik Universität Karlsruhe Volker Büge, Yves Kemp, Marcel Kunze, Günter Quast, Armin Scheurer LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  2. Outline • What is Particle Physics / High Energy Physics? • Introduction • Computing and Storage • Current Experiments • Introduction to Grid Technologies • What is a Grid? • The Worldwide LHC Computing Grid • Live Demo • Virtualisation in the Grid Environment • Hardware Consolidation • Virtualisation and Batch Systems • Gathering Resources from idle Desktop PCs • Conclusion & Outlook LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  3. Macrocosm 101 m What is Particle Physics? - Dimensions Crystal Molecule Atom 10-2 m 10-9 m Quark / Proton Nucleus Electron 10-10 m <10-18 m 10-15 m 10-14 m LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  4. What is Particle Physics? - Big Bang Today Heavy Atoms ... Light Atoms Energy Timescale Nuclei Nucleons ... Elementary Particles Big Bang LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  5. Our Instruments – Accelerators - LHC Lake Geneva Large Hadron Collider Circumference: 27 km Beam energy: 7 TeV Below surface: 100 m Temperature: -271 °C Energy use: 1 TWh/a 4 large experiments: CMS ATLAS LHCb ALICE CERN Airport LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  6. Our Instruments – Detectors - CMS Compact Muon Solenoid - CMS Specifications: total weight: 12 500 T overall diameter: 15 m overall length: 21,5 m magnetic field: 4 Tesla LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  7. Our Instruments – Detectors - Event Display The collision of 2 high-energetic hydrogen nuclei (protons) produces several 1000 particles LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  8. Our Instruments – Detectors - Data Rates Reduction with ASICs ~ 60 TB/sec Level 1 Trigger Collision Rate: ~ 40 MHz ~ 150 GB/sec Event size: ~1.5 MB for Offline- Analysis High Level Trigger ~ 225 MB/sec Tape & HDD Storage Software Data Reduction (PC Farm) LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  9. Our Instruments – Detectors - Storage & CPU Resource expectations of the 4 LHC experiments LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  10. CMS 38 Nations 182 Institutions 2000 Scientists & Engineers CMS High Energy Physics Collaboration LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  11. CMS Software Framework & OS • long-term experiments, large fluctuations of collaborators • transparency for analyses • huge purpose-designed software framework (~ 1GB) • Some OpenSource projects (ROOT Analysis Framework, GEANT for particle interactions in matter, ...) • we build our own read-out hardware drivers, ... only reasonable answer OpenSource, Linux LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  12. Peculiarities of HEP Data Processing Static program and dynamic data: e.g. meteorology, geography, finance • set of fixed programs used to analyse new data in short intervals • same approach for SETI@home Static data and “dynamic code”: e.g. in High Energy Physics • data acquisition very expensive • data is analysed repeatedly with iteratively improved code • e.g. ~1000 publications from 500 people with ~1TB preprocessed data! LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  13. Data Storage and Access Constraints and Approaches: • HEP experiments very expensive! • redundant storage of 1.5 PetaByte per year only for CMS! • not possible at one single computing centre (founding constraints) • distribution of data to participating computing centres all over the world • huge datasets (~TeraByte) cannot be transferred to each user • the analysis job goes “where the data set is” • ensure access to these data for more then 2000 physicists from 182 institutes in 38 countries (in CMS) • access to data and computing resources without local login for the user The LHC experiments cope with these challenges using grid technologies – The Worldwide LHC Computing Grid LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  14. What is Grid Computing? Definition of Grid Computing by Ian Foster: • coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations • the ability to negotiate resource-sharing arrangements among a set of participating parties (providers and consumers) and then to use the resulting resource pool for some purpose. Today, grids are used in science to … • enable research and collaborations independent from the geographical location • share distributed resources like a single computing cluster or access to storage resources • load balance of resources – opportunistic use! LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  15. The Worldwide LHC Computing Grid (WLCG) The WLCG Computing Model: • computing centres are organised in a hierarchical structure • different policies concerning computing power and data storage The tiered architecture: 1 Tier0 (at CERN): • accepts raw data from detectors • data transfer to Tier1 centres 8 Tier 1 centres: • secure data archiving • reconstruction & reprocessing • coordination of associated Tier2s Several Tier 2 centres: • capacity for data-intensive analysis • calibration & Monte Carlo simulation LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  16. The WLCG - Members Currently 189 sites participating LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  17. The WLCG - Basics The basic concepts: Authentication: Who am I ? • concept of certificates • “equivalent” to a passport, ID card etc. Authorization: What may I do ? • certain permissions, duties etc. • “equivalent” to a visa or access list • Virtual Organisation Membership Service User can adapt several different roles, e.g. software manager, normal user, … LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  18. The WLCG - LCG Middleware I VO Server • registry office of a VO • contains all users and their roles within a VO Grid-wide services: Resource Broker • “intelligence” of the grid • distributes incoming job requests to matching resources Information Service • collects and publishes information on resources connected to the LCG LCG File Catalogue • global file index for a Virtual Organisation LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  19. The WLCG – LCG Middleware II Site-wide services: • User Interface • access point for the user to the grid • Computing Element • portal to the local batch system of a site • Storage Element • offering disk space to a VO • portal to the local storage • Monitoring Box • collects and publishes information on grid jobs executed at a site LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  20. The LHC Computing Grid – Access A job’s way through the grid: LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  21. Submission of the following script: #!/bin/tcsh –f hostname –f whoami uname -a LiveDemo! The WLCG – Live Demo I The job is described in one configuration file, containing: • specification, which resources are required, for example • names of special sites to prefer/ignore • special software release installed • minimum CPU-time of a queue • names of all files which • the job need to be executed on the site (InputSandbox) • the user wants to get back when the job has finished (OutputSandbox) LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  22. The WLCG – Live Demo II The job configuration file: JobDescription.jdl Executable ="MyScript.csh"; StdOutput ="std.out"; StdError ="std.err"; InputSandbox ={"MyScript.csh"}; OutputSandbox ={"std.out","std.err"}; VirtualOrganisation = "cms"; 255 submissions of this job have been executed at these sites: LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  23. Scientific Linux – The Science Linux Flavour! What is Scientific Linux? • large computing facilities in HEP have run adapted RedHat distributions • change in the RH policies leads to expensive licences • CERN, other labs (also non-HEP) and universities use a recompiled RH Enterprise Server as base distribution: In our case: Scientific Linux CERN Edition • current release: Scientific Linux 3 with a 2.6 Kernel • recompiled RedHat Enterprise Server 3 • support and updates provided by CERN • optimised for HEP environment • no fee for licenses • will change to SLC4 in autumn LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  24. Virtualisation I Para Virtualisation, e.g. XEN • different hardware components are not fully emulated by the host OS. It only organises the usages  Small loss of performance • layout of a Xen based system: Privileged host system (Dom0) and unprivileged guest systems (DomUs) • DomUs are working cooperatively! • guest-OS has to be adapted to XEN (Kernel-Patch), but not the applications! LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  25. Both CPUs in the native OS used for one compilation Only one CPU in the VM Slightly smaller performance of the Xen based VM compared to the native OS. Virtualisation II Standard application benchmark: Linux kernel compilation (4 in parallel; make –j4) LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  26. CE SE MON Host (XEN) CE SE MON Hardware Consolidation at a WLCG Tier3 Centre I Typical situation at a university’s Tier 2/3 centre: • for reasons of stability we recommend to run each service in an isolated OS instance. • varying load on the different machines • no full usage of resources • “recycling” of older machines leads to a heterogeneous hardware structure • high administrative effort for installation and maintenance of the system Virtualisation of these machines lead to one single machine to be maintained and to homogenous OS installations LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  27. Hardware Consolidation at a WLCG Tier3 Centre II Advantages through virtualisation: • a reduction of hardware overhead : Only one single high- performance machine needed for the complete LCG installation including a test WN cheaper and easier to maintain • easy and fast setup of basic OS by copying VMs image files • possibility of migrating VMs to other machines and backup • cloning of VMs before upgrades of LCG to enable tests • less service interrupts and a more effective administration • balanced load and efficient use of the server machine • interception of CPU peaks LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  28. Hardware Consolidation at a WLCG Tier3 Centre III Realisation of a full LCG environment in Virtual Machines • host system with Virtual Machine Monitor (VMM) Xen • OS: Scientific Linux 4 ( native with 2.6 kernel) • CPU: Intel(R) Pentium(R) 4 CPU 3.0 GHz • Memory: 1GB • VMs: CE, SE and MON now run on SLC3 • second LCG installation for testing purposes available • both environments for LCG 2.7.0 fully integrated into our batch and storage system Complete Tier 3 infrastructure on one machine works LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  29. Virtualisation of Batch Queues Basic Ideas: • Different groups at the same computing centre need different Operating Systems • Agreement on one OS or no resource sharing • virtualisation allows to dynamically partition a cluster with different OS • each queue is linked to one type of Virtual Machine  Such an approach offers all advantages of a normal batch system combined with the free choice of the OS for the computing centre administration and user groups! LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  30. Dynamic Partitioning of a Cluster I • Requirements • should be independent from batch system server and scheduler • no modifications on existing products • flexibility through a modular structure • Implementation: • a daemon is observing the queue and keeps track on next jobs which will start according to its priority • will start VM with desired OS and register it to the batch system • keeps track of used and unused Host-Nodes • Peculiarities: • optimise number of shutdowns and restarts of VMs • concept should not affect the prioritisation of the scheduler! LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  31. Running Empty Running Suse10 SLC3 Batch Server Which OS is required next? daemon 2. Starts requested VM 3. VM is connected to batch system Worker Node Virtual Machine Dynamic Partitioning of a Cluster II Test System: Simulation of a cluster with 19 nodes • 2 Dual Opteron Machines with 2GB RAM each • each is hosting 10 VMs • 1 Torque server with MAUI scheduler, running the daemon • 19 Virtual Computing nodes LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  32. What about idle desktop PCs ? High-performance Desktop PCs often not used for hours, for example: • computing pools at universities outside lectures • office PCs at night • many more! • Use this power for analyses – Condor cluster for jobs that are independent of the OS VM on desktops would offer: • dedicated development system for different groups of users • environment for OS dependant analyses tasks Starting required OS in VMs on idle desktop PCs to harvest this computing power LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  33. Conclusion & Outlook • High Energy Physics experiments • collaborations are international • OpenSource principles indispensable in HEP collaborations • need large storage and computing resources • cope with these challenges using grid technologies and Linux • Virtualisation • hardware consolidation at a Tier 2/3 centre • dynamic partitioning of shared batch system with different OS • opportunistic use of idle desktop PCs Linux with XEN allows to introduce an new layer of abstraction into the Grid  gets more flexible and stable! LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  34. Links Cern http://www.cern.ch/ The Large Hadron Collider http://lhc.web.cern.ch/lhc/ CMS http://cmsinfo.cern.ch/Welcome.html ROOT http://root.cern.ch GEANT http://wwwasd.web.cern.ch/wwwasd/geant/ Worldwide LHC Computing Grid http://lcg.web.cern.ch/LCG/ Enabling Grids for E-sciencE (EGEE) http://public.eu-egee.org/ LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  35. Links GOC Grid Monitoring http://goc02.grid-support.ac.uk/googlemaps/lcg.html Scientific Linux https://www.scientificlinux.org/ Scientific Linux Cern http://linux.web.cern.ch/linux/ Global Grid User Support https://ggus.fzk.de/ The Xen virtual machine monitor http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ Institut für Experimentelle Kernphysik – University of Karlsruhe http://www-ekp.physik.uni-karlsruhe.de/ Institute für Wissenschaftliches Rechnen – Forschungszentrum Karlsruhe http://www.fzk.de/ LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  36. Backup Slides LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  37. Our Instruments – Accelerators - LINAC PEP II Rings Positron Electron Project SLAC Linear Accelerator Length: 3,2 km Beam Energy: 50 GeV Some experiments: SLC GLAST B Factory BaBar SLAC Stanford Linear Accelerator Center SLAC LINAC LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  38. The WLCG - CMS Resources I In total: 11019 CPUs LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  39. The WLCG - CMS Resources II In total: 1939 TeraBytes LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

  40. Virtualisation I Para Virtualisation, e.g. XEN • different hardware components are not fully emulated by the host OS. It only organises the usages  Small loss of performance • layout of a Xen based system: Privileged host system (Dom0) and unprivileged guest systems (DomUs) • DomUs are working cooperatively! • guest-OS has to be adapted to XEN (Kernel-Patch), but not the applications! LinuxTag 2006, Wiesbaden Volker Büge 6.Mai 2006

More Related