1 / 17

Data processing infrastructures @ LIP-Lisbon

Data processing infrastructures @ LIP-Lisbon. Luso, 12th January 2008. Gonçalo Borges Grid Computing Team LIP-Lisbon. Testbeds @ LIP-Lisbon. Local test & development infrastructure Virtual machine testbed Aimed to LIP/Grid admins experiencies Pre-Production infrastructure

jogle
Download Presentation

Data processing infrastructures @ LIP-Lisbon

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data processing infrastructures @ LIP-Lisbon Luso, 12th January 2008 Gonçalo Borges Grid Computing Team LIP-Lisbon

  2. Testbeds @ LIP-Lisbon • Local test & development infrastructure • Virtual machine testbed • Aimed to LIP/Grid admins experiencies • Pre-Production infrastructure • Test EGEE middleware releases • Aimed for testing VO applications & grid middleware deployment and performance • Production infrastructure • EGEE, Int.EU.Grid, LIP FARM • Where grid users and LIP users run their applications goncalo@lip.pt LIP Grid Computing

  3. VANESSA: Virtual machine testbed • After MONICA and AMANDA... • then it came VANESSA! • VANESSA: A physical machine supporting a virtual testbed • 4 VMs implemented via Xen technology and “LVMs partitions” • The images of the VMs can be recorded at any moment in time • OS/Software breaking is allowed • Used to develop new grid software • SGE integration with EGEE middleware • Used to test grid middleware and applications • Under different OS and environments goncalo@lip.pt LIP Grid Computing

  4. Pre-Production Testbed • Small EGEE Grid Cluster • 1 Computing Element; 1 Storage Element • DPM with ~100 GB of storage space • 6 CPUs on the Worker Nodes • Intel Pentium IV 2.0 GHz, Intel Celeron 2.4 GHz • Used to test Grid middleware • After middleware certification and before deploying it in EGEE production infrastructure • Tries to simulate a real production enviroment • Same services as production, real users submitting jobs, ... • Middleware updated or re-installed almost every week • Also really used for important tasks • DILIGENT Press Release goncalo@lip.pt LIP Grid Computing

  5. LIP Production Testbed / LIP Farm • LIP is envolved in 3 different grid projects • EGEE, Int.EU.Grid and EELA. • Support to more than 17 VOs • EGEE: atlas, auger, biomed, cms, compass, dteam, ops, swetest • Int.EU.Grid: ibrain, icompchem, imain, iplanck, itest, itut,iusct • EELA: edteam, eela • LIP groups giving resources to the LIP FARM • calo, cms, pet, compass, calo • Total of Resources = 88 cores • EGEE and Int.EU.Grid middleware installed everywhere • All machines may work as Grid Worker Nodes • Just a matter of configuration at the batch system level goncalo@lip.pt LIP Grid Computing

  6. Int.EU.Grid CE (SGE sub) EGEE/EELA CE (SGE sub) lnlip01.lip.pt (SGE sub) WNs (SGE exec hosts) Resources architecture sge01.lip.pt (SGE Qmaster) goncalo@lip.pt LIP Grid Computing

  7. Groups & resources • Grid and local queues • Most machines: Dual core AMD Opteron 2.2 GHz goncalo@lip.pt LIP Grid Computing

  8. Managing queues • By default all jobs run in i386 queues • Unless a x86_64 queue is requested at submission time • slowlip_on_i386 • Very short time queue • Just to test if your application is running • Have to explicitely ask for it at submission time • fastlip_on_i386 • ~ 1/2 of the LIP IT machines • The other 1/2 is reserved to the grid • ~ 1/2 of each group machines • The other 1/2 is reserved to the group • fastlip_on_x86_64 • ~ 1/2 of each x86_64 group machines goncalo@lip.pt LIP Grid Computing

  9. calolip (seq #0) compasslip (seq #0) fastlip_on_i386 (seq #1) Scheduling policies for LIP users • LIP users start by filling their group machines • When filled, start to fill other group shared machines • The next job to be run in a group machine is always from a group member, if there are group members pending jobs... • Sharing policies between groups members goncalo@lip.pt LIP Grid Computing

  10. Usage (Number of submitted jobs) Max = 33093 jobs; Total = 158125 jobs goncalo@lip.pt LIP Grid Computing

  11. LIP usage (Number of submitted jobs) Max (Grid) = 3841 jobs goncalo@lip.pt LIP Grid Computing

  12. Max (Grid) = 72043 hours; Total = 173601 hours Usage (Cpu Time) goncalo@lip.pt LIP Grid Computing

  13. Max (Grid) = 11402 hours LIP usage (Cpu Time) goncalo@lip.pt LIP Grid Computing

  14. Usage (Cpu Time/Job) goncalo@lip.pt LIP Grid Computing

  15. Usage (Memory/Job) goncalo@lip.pt LIP Grid Computing

  16. Usage (“Efficiency”) goncalo@lip.pt LIP Grid Computing

  17. “O organismo humano expele, diariamente, entre 2 a 3 litros de àgua, seja pela transpiração, pela respiração ou pela urina.” • “97% são águas salgadas; 2% são águas doces; 0,99% são águas doces subterrâneas e na forma de vapor. • Restam 0,01% de água doce disponível... grande parte já poluída!” • “Uma pessoa com peso e saúde normal pode viver até 30 dias sem comida. Essa mesma pessoa poderá morrer em 72 horas caso não possa beber água.” “Talk” que mais àgua meteu! goncalo@lip.pt LIP Grid Computing

More Related